Currently, we are using Databricks Repo on Databricks with Azure Devops as .git.
Azure Devops is configure to deploy new update on each branch [dev,main,release] using
databricks repos update --path /Repos/dev/my_repo --branch dev
databricks repos update --path /Repos/staging/my_repo --branch main
databricks repos update --path /Repos/prod/my_repo --branch release
It's working well.
We are having issues to define some proper process to unit_test our notebook, specially 'staging notebooks from Azure DevOps pull request', before accepts it.
Since we are using only notebook, we have to push our "staging pull request code' in Databricks, then run unit_tests.
This is the CI workflow I would love to implement:
Create a repo for "unit_test" in /Repos/test/my_repo
User create a feature_branch and create pull request to dev.
During Azure Devops CI, after resolving conflicts, update '/Repos/test/my_repo' with fresh merged code in AzureDevops Pull request using
databricks repos update --path /Repos/test/my_repo --branch 'merged branch from pull request'
then, trigger Databricks Worflows associated to that Repo /Repos/test/my_repo which run unit tests.
If unit-tests are OK from Databricks Workflows execution, then accept Pull Request on AzureDevops, and do an update of dev branch.
I did look at AlexOtt example (https://github.com/alexott/databricks-nutter-repos-demo/blob/master/azure-pipelines.yml) which deploy 'sourceBranch' to test, and not 'sourceBranch' + 'targetBranch'
Is there a way, in AzureDevops, to access 'pull request staging code' as git branch name or tag, to be able to push it on Databricks ?
This is a quick stage on AzureDevops to explain my need :
- stage: Unit_test_PR_on_databricks
condition: and(eq(variables['Build.Reason'], 'PullRequest'), eq(variables['System.PullRequest.TargetBranch'], 'refs/heads/dev'))
jobs:
- job: unit_tests
displayName: "Unit tests"
pool:
vmImage: ubuntu-latest
steps:
- script: |
echo "Testing code from PR on Databricks"
# THIS SHOULD UPDATE test repo with
databricks repos update --path /Repos/test/my_repo --branch "$(branchName)"
# THIS SHOULD TRIGGER WORKFLOWS WITH UNITTESTS
databricks jobs run-now --job-id ${job_test_workflow}
displayName: 'Trigger unit test on Databricks'
I don't want to accept any pull request on [dev/main/release] without having some unit_tests executed and success on Databricks.
Currently, with all documentations, it seems that the only way using Databricks Repos is :
accept PR
deploy just accepted PR to Databricks and run unittests.
If fail, do a new PR with fixes.
With that kind of process, git branches will never be really clean (specially dev), and many 'fix pull request' if any unit tests fail after merging.
Related
trigger:
branches:
include:
- master
tags:
include:
- Showcase
pr:
branches:
include:
- master
the code above is from my yaml azure pipeline with using bitbucket repo. Anytime I do a PR in my bitbucket repository I expect build pipeline to run automatically in azure devops but it's not the case. Currently, I have to run the pipeline manually to complete PR validation. creating pipeline with the same line of code under another repo works correctly. I'm not sure what the problem is.
I created anoter dummy pipeline with a new project, the pr triggered build automatically when pr to the the master. the webhohook seems to be working fine. the trigger option doesn't override the yaml pipeline. I can't think of anything else to check.
I'm trying to implement azure automatic pipeline build with azure-pipelines.yaml
i have tried the below methods in order to get it work.
azure-pipelines.yaml(yml)
.vsts-ci.yml(yaml)
.azure-pipelines.yaml(yml)
But when I'm trying with UI it's detecting.
Is it possible to build auto build pipeline in Azure-devops?
If you're building a GitHub repo in an Azure DevOps pipeline, there's some stuff you need to setup.
Azure Pipelines can automatically build and validate every pull request and commit to your GitHub repository. This article describes how to configure the integration between GitHub and Azure Pipelines.
If you're new to Azure Pipelines integration with GitHub, follow the steps in Create your first pipeline to get your first pipeline working with a GitHub repository, and then come back to this article to learn more about configuring and customizing the integration between GitHub and Azure Pipelines.
Source: Build GitHub repositories.
If you explicitly need a DevOps Pipeline to build your GitHub repo, have a look at the article linked to above. Otherwise think about either working with GitHub Actionshttps://github.com/features/actions on your GitHub repo, or hosting your repo in Azure Repos.
Depending on where your repository is located you should place your .yml file inside this repository in order to work with the trigger. You should also use the trigger keyword.
Then when you create your pipeline, you will choose Github, Azure Repos Git, whatever you use and select existing pipeline.
If you use the starter pipeline of Azure Devops, the pipeline will trigger out of the box when a commit is pushed on the main branch.
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
- script: |
echo Add other tasks to build, test, and deploy your project.
echo See https://aka.ms/yaml
displayName: 'Run a multi-line script'
You can change main to which ever branch you need to trigger automatically.
I have a git repository that has been imported to azure repos but when I make a new commit it doesn't reflect the new commit on azure, is there some config I need to enable to allow this?
Sorry if the question seems dumb but I have been navigating through the UI and I can't see how to achieve this
thank you for all your help
There are many ways to achieve your needs. You can refer to the sample I implemented using Azure pipelines(I assume your repository is imported from GitHub to azure repos):
1.Create a pipeline and choose your GitHub repo as source:
2.Here is my configuration of the yaml file:
trigger:
- '*'//This pipeline will run whenever you push an update to the GitHub repo
pool:
vmImage: 'windows-latest'
steps:
- task: gitmirror#0//You can install this task in marketplace
inputs:
GitRepoUrl: 'https://{PAT}#dev.azure.com/{Organization Name}/{Project Name}/_git/{Repo Name}.git'//This is the URL of Azure Repo, you need to add a personal access token in the URL
3.Now, when you update the files in GitHub, it will trigger the pipeline in Azure DevOps and then update your Azure repository.
I am working on an Azure pipeline for a dotnet core project that runs on Windows Self hosted agent.
I want my pipeline to use Git commands to check out the release branch and merge the develop branch into it. Next steps will be to build the release branch and deploy to intranet servers
I don’t know Git wording very good, I was using TFS for years. I use the commands below and got the logs here:
- task: CmdLine#2
displayName: Checkout Release branch
inputs:
script: |
#echo off
git checkout release
git pull develop
git status
From the logs, I understand:
It downloads the content of the develop branch because it is the default branch in GitHub, I’d rather want the release branch but I believe Azure is like that
I manage to switch to release but I have these errors that I don’t understand:
##[error]Previous HEAD position was bc94bd2 Update Staging Build Pipeline.yml
##[error]Switched to branch 'release'
I understood that pull can be used with local or remote branch so I use it to fetch and merge the develop branch to the release branch but I get: [error]fatal: 'develop' does not appear to be a git repository
Do I have to specify credentials on every calls to git?
On the last step, it fetches again the code from the develop branch and I understand why
If you could help me improve my script, that would be great,
Many thanks.
You can use git merge commands to merge branches. To merge develop branch into release branch you can use git merge origin/develop. Check the document for more information. See below example:
steps:
- checkout: self
persistCredentials: true
- task: cmdLine#2
inputs:
script: |
#echo off
git checkout release
git merge origin/develop
git status
However, it is not recommended to deploy release branch in above way. You can change the default branch of your azure pipeline to release branch and enabled the Continuous Integration trigger for release branch.
So that you can create a pull request to merge develop into release from the github UI or by using commands. After develop is merged into release, the azure pipeline will be automatically triggered to deploy from release branch. Note: the azure pipeline yaml file must exist in release branch too. See below steps:
1, To change azure pipeline branch from develop to release:
On your azure devops pipeline Edit page, Click the 3dots and click Triggers
Go to YAML tab--> Get Sources-->Click the 3dots to change the default branch.
2, Set CI trigger for release branch
In the azure pipeline yaml file, set the trigger to include release branch(You can also set PR trigger):
(Actually you do not need to follow above steps to change the default branch. You just need to include the azure pipeline yaml file in release branch and set the CI trigger to include release branch as below)
trigger:
branches:
include:
- release
exclude:
- develop #if you want disable CI trigger for develop branch
By adding the CI trigger to include the release branch in the azure pipeline yaml file. Azure pipeline will automatically be triggered on release branch when merging from develop into release branch.
I am setting up CI/CD at work and there is one step I’m not sure how to do and furthermore, if it is a right thing do.
For background, I am used to develop in C# with Visual Studio, source code in TFS and deploying with basic script that copies files on the intranet.
Now, I’m requested to setup Build and Release pipelines on Dot Net Core projects in GitHub.
I have three branches on this project: DEV, RELEASE and MASTER
I created one pipeline that triggers on DEV’s commits, creates an artefact and deploy to DEV server.
Those are the pipelines that deploy all developers work to a DEV server where they run their own tests.
Next step, when we want to deploy to staging servers, we click a button in Azure, this merge the DEV branch to the RELEASE branch but I know close to nothing in GitHub, not even sure those are the appropriate words.
When the merge is done, this will trigger a build pipeline that will create a different artefact, when this artefact is updated, deploy to staging server.
Once this release is validated on Staging and Quality, we would merge RELEASE to MASTER and do the same until PROD servers. It is all on intranet and self-hosted agents.
Is that a good way of doing things? Can it be done this way? I need a PowerShell task or is there something that exists?
If you are using Azure DevOps pipeline, the pipeline should select GitHub for repository type, then we can configure the CI trigger.
a. Configure CI trigger:
Classic steps:
1.Open project setting->Service connections->select GitHub-> create a new GitHub service connection
2.Create a new build pipeline via classic editor-> Select GitHub as the source.
3.Open pipeline->select the tab Triggers-> enable the option Enable continuous integration and configure the Branch filters
b.YAML steps:
1.Open project setting->Service connections->select GitHub-> create a new GitHub service connection
2.Create a new build pipeline and select GitHub(YAML)
c.The sample of Check out GitHub repositories in your pipeline
resources:
repositories:
- repository: MyGitHubRepo # The name used to reference this repository in the checkout step
type: github
endpoint: MyGitHubServiceConnection
name: MyGitHubOrgOrUser/MyGitHubRepo
trigger:
- {branch name}
Configure CD trigger:
Please refer to this doc to configure the release trigger.
If you are using GitHub action.
Please select the correct workflow to configure the CI/CD, Please refer this doc for more details
CI sample:
on:
push:
branches:
- ' DEV'
Update1
When Dev branch is updated, it is built and deployed to Dev server by pipeline.
Create build A and release B, configure the CI build trigger, when the branch Dev is updated, it triggers the build pipeline A, and when build pipeline A is completed, it triggers the release pipeline B.
click a button to synchronize Dev branch to Release branch.
We cannot see the button, as a workaround, we can add task powershell and call the API to create pull request and complete the pull request. We also can add task cmd and publish the code via git cmd