I have my code in a Azure DevOps repos., and use Azure Databricks to execute and develop my code.
I am using the Repos functionality in Databricks where I can connect to my code repos, commit and push code to my branch in DevOps.
When creating a new branch from Databricks it is created with no problem, and I can see in DevOps that the branch is created with me as owner.
I am though having a problem when committing/pushing my changes from the Databricks interface. My pushes show up in DevOps without any user, and just shows a empty user.
Has anybody experienced this, and have a fix for it ?
This was a know issue by Databricks. I've reach out to them and they have a fix on the way.
Related
My wish is to have the contributions I make to a repo in Azure DevOps show up on my GitHub profile.
Is there a way to link this up properly?
The email I use for credentials when authenticating towards Azure DevOps is different than my GitHub connected email.
I have tried to use the same credentials for both but to no avail. Maybe I'm missing something.
Hopefully, someone out there has got this working.
Thanks a bunch in advance!
Firstly, you should understand that GitHub and Azure DevOps are independent systems. Azure DevOps has provided some functions e.g REST API,UI function to get repository from GitHub. But It’s a one-way data transfer from GitHub to Azure DevOps.
There is no supported REST API or UI functions that can sync repo changes from to Azure DevOps to your GitHub according DevOps documents at present.
I am afraid that your GitHub contributions graph can’t get credited when you push a change to the Azure DevOps repo.
I am having all my codes in Azure Databricks notebook. I want to automate execution of my notebook using azure devops.
I have linked my Databricks Repos with Azure Repos. Created a build pipeline to create artifact of my code. In release pipeline.
I was able to deploy my code to lower environment(Here same Databricks workspace where I have my code).
But I was not able to execute/Run notebook directly from Azure Devops.
How do I deploy code from my lower environment to higher environment(Prod Server) Databricks workspace.
To deploy code to higher environment Is it same as deploying to lower environment. Just changing details of workspace.
Like Databricks Token, Workspace URL, etc.
You need to create release pipeline and push your code to GIT repo. Follow this Microsoft documentation for details -
https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/ci-cd-azure-devops#:~:text=The%20release%20pipeline%20deploys%20the,Pipelines%20menu%20and%20click%20Releases.
This demo shows how to do that. The steps include:
Testing
Update checkout at Databricks workspace - for example, you may have a separate checkout for testing. You can do that via Repos REST API, or via databricks-cli (databricks repos update command)
Triggering execution of tests by using the Nutter library
Reporting testing results to DevOps
Release
If tests are successful, you can push updates to staging/production environment - this could be done via DevOps Release pipelines
You can use the same Repos REST API or databricks-cli to update checkout in staging/production
My organization recently changed from using personal GitHub accounts to using GitHub Enterprise Managed Users. Since that change our pipeline continuous integration (in Azure Devops) has stopped working. We can still run the builds manually without issue, but it isn't even attempting to start the builds automatically. I've created new personal access tokens for the pipelines, but that hasn't fixed the continuous integration. Does anyone have any suggestions on what I should look at to get this working again?
Instead of relying on user-authenticates service connections between GitHub and Azure Pipelines, I recommend switching over go the Github App for Azure Pipelines to control what repositories Azure Pipelines can access:
https://github.com/apps/azure-pipelines
That will drastically simplify access maintenance between GitHub and Azure Pipelines and will break the reliance on personal or enterprise managed accounts.
I wound up fixing this by going to the repository in Git Hub, go to settings on the repository, and then to Web Hooks. In there, the hook that Azure Pipelines uses had an error on it because when our tech group migrated the repo they did not bring the secret over. To fix it I just had to delete the web hook, go to the pipeline and a button appeared to restore the connection.
I have used Azure DevOps 'Import Repository' to clone a repo from GitHub so that I can maintain a personal copy. I don't see any way to maintain the remote connection to GitHub so that I can merge in upstream changes.
Does Azure DevOps have such functionality or will I need to do this merging manually/locally?
No there is no way to treat Azure DevOps as a fork for GitHub. You need to manually perform merges and push/pull to and from both GitHub and Azure DevOps.
Not sure why you'd want to use Azure DevOps for your personal clone. As an alternative you can use GitHub for source code repository and link Azure Boards to GitHub and use Azure Pipelines to build, if those are your primary reason to use Azure DevOps.
Hi Databricks and DevOps gurus,
I want to connect a Databricks Notebook and all of its versions/revision history with DevOps.
I follow the Microsoft documentation and just before the end I get an error in Databricks:
First I thought maybe I am not providing the right link but after I click "Create PR" it leads me to my DevOps environment(so the link is the right one) and it asks me to make a new pull request.
As you can see my only branch at the moment is master.
I am not new to Databricks, but I am new to DevOps and version controlling, and I don't know any way of proceeding from here. Any help would be highly appreciated.
If you see this error Error while listing Git repo branches: Failed to retrieve repo branch list: com.databricks.webapp.versioning.BadAzureDevOpsCredential:... It is because your azure devops services organization and the azure databricks are not in the same Azure AD tenant.
See below extract from document here.
Authentication with Azure DevOps Services is done automatically when you authenticate using Azure Active Directory (Azure AD). The Azure DevOps Services organization must be linked to the same Azure AD tenant as Databricks.
See the similar reported issues here and here for more information.