Azure Data Factory supports organizing it's objects (Pipelines, datasets etc) into Folders with in the ADF Studio.
When the ADF is connected to a Git repository all the ADF Objects are moved into git based on the AD Component Type.
Below is how the Pipelines are organized with in the ADF Studio
After saving the changes, this is how the files are structure in the git
If I organize the P1-Pipeline.json and P2.Pipeline.json into separate folders with in git then the ADF Studio doesn't recognize them.
It is really important for us to organize the Pipelines (and other ADF Components) into folders within git as each Folder represent a separate team.
Is there a possibility to organize the ADF components into git folders? If not, what is the recommended approach for selective deployment of the ADF components
If I organize the P1-Pipeline.json and P2.Pipeline.json into separate folders with in git then the ADF Studio doesn't recognize them.
No, you shouldn't change the location of the pipeline.json file to another folder in Git Repo. You should create any pipeline in a separate folder and if want to move it to another folder, move it in ADF Studio only.
By clicking on the three dots in front of the pipeline name, click on Move item option and then choose the folder in which you want to move the pipeline.
In the below example I have moved the pipeline1 from project1 folder to project2 folder.
And I can still see both the ARM files of these pipelines in Azure Repo.
If you want to keep each pipeline in a separate folder, it’s better to create ARM template for each pipeline by clicking on Save as template option, refer below image.
Your each ARM template as json will be stored in separate sub-folder by pipeline name under main templates folder as shown in below image.
Related
I would like to schedule (for my company) a backup of our most important data in Azure DevOps, and that, for different reasons : security, urgent recovery required, virus, migration, etc...
I can execute a backup of the repositories and the Wikis (because it's under GIT, so easy to download), but how can do a backup of the "Board" section (Backlogs, Work items, etc...), and the build pipelines definitions?
How to backup the data on Azure Devops?
In current Azure DevOps, there is no out of the box solution to this. You could manually save the project data through below ways:
Source code and custom build templates: You can download your files
as a zip file. Open ... Repository actions actions for the
repository, file, or folder and choose Download as Zip. You can
also Download from the right side of the screen to download
either all of the files in the currently selected folder, or the
currently selected file.
This process doesn't save any change history or links to other
artifacts.
If you use Git, clone your repositories to retain the full project
history and all the branches.
Build data: To save logs and data in your drop build folders, see
View build results.
Work item tracking data: Create a work item query and open it using
Excel. Save the Excel spreadsheet.
This process doesn't save any attachments, change history, or links
to other artifacts.
build/release defintions: you could export the json file for them and then import them when restoring them.
There has been a related user voice, you could monitor and vote up it: https://developercommunity.visualstudio.com/content/idea/365441/provide-a-backup-service-for-visual-studio-team-se.html.
Here are some tickets(ticket1 ,ticket2) with the same issue you can refer to.
If you want to create scheduled tasks, you can write a script by using the Azure CLI with the Azure Devops Extension
As you said, for the repositories, it's quiet easy as they are Git repositories.
I wrote such a script that we could improve to also backup the Workitems, Backlog, etc...
It's open source, let me know what you would like to backup first and I'll improve it.
Github : azure-devops-repository-backup
Is there a way to generate a solution and project file out of a folder structure through a azure pipeline .ymal stage?
The way the project has been set up is that there are lots of other .git repos set up inside a master repo and inserted though subtrees. These repos don't have a .sln in themselves but instead when they are added into Unity they get added into the projects .sln and a .csproj is generated for each of the assemblies within the submodule (package)
What I'm looking to do is to have documentation generated for each of these submodules whenever an update is pushed to its master (not the projects it lives in master) as these tend to be more utilities and self contained systems. Problem I'm facing is that I can trigger all the documentation system with docFX but because this module does not contain a .csproj I'm unable to generate the documentation for it. so I'm wondering if its possible to have a step where I can create a project file for all scripts that are within a folder structure, and as such then have a project file for docFX to work of.
I know its not ideal in any sense, but wondering if its a possibility while I investigate further into other solutions.
Is there a way to generate a solution and project file out of a folder
structure through a azure pipeline .ymal stage?
For this issue, I am afraid that azure pipeline is impossible to achieve this.
".csproj" is a Visual Studio .NET C# Project file extension. This file
will have information about the files included in that project,
assemblies used in that project, project GUID and project version etc.
This file is related to your project. It will be automatically
generated when we create
".sln" is a structure for organizing projects in Visual Studio. It
contains the state information for projects in .sln (text-based,
shared) and .suo (binary, user-specific solution options) files. We
can add multiple projects inside one solution.
Azure pipeline cannot generate a solution and project file according to the folder structure.
This is a TFVC repo in Azure, not Git. It is running in Azure DevOps Services, not local in Azure DevOps Server (2019). This is a classic pipeline, not YAML.
I got as far as adding a variable that contains the Label value I am looking to package into the zip file.
I can't figure out how to get the sources by Label value. In the Pipeline Get Sources step, I've narrowed the path down, but then I need to recursively get source files that have the Label in the variable I defined.
The next step is to zip those source files up, I've added an Archive task to which I will change the root folder from "build binaries" to the sources folder.
This is necessary for this particular project because we must pass the source files to the vendor as a zip for them to compile and install for us. The developers create/update the source files, build and test them locally, then apply a Label to the sources for a given push to the vendor.
When configuring 'Get sources' step, there is no any option or method that can only map the source files with the specified label.
As a workaround, in the pipeline job, you can try to add the steps to filter out the source files with the specified label, and use the Copy Files task to copy these files to a folder, then use the Archive Files task in this folder.
[UPDATE]
Normally, a pipeline run will automatically check out the file version (changeset) that triggers the run. If manually trigger the the pipeline, by default the run will check out the latest changeset if you do not specify one.
The labels are used to mark a version of a files or folders, so you also can get the specific version of files or folders via the labels.
In your case, you can try using the 'tf get' command to download the files with the specified labels.
Is there a way to download all files with a specific name from the master branches of all the projects in an Azure DevOps installation?
I have been tasked with documenting all of the entries in all of the appsettings.json files in our entire codebase and I would prefer to not have to go through all 300 repositories to manually download these files if I don't absolutely have to.
Is there a way to download all files with a specific name from the
master branches of all the projects in an Azure DevOps installation?
This is not supported. Please check this document, we recommend different project(with one or more repos) for different products/sub-modules of big product. So in Azure Devops Service there's no such out-of-box feature to find/download files across projects.
A possible direction:
If those appsettings.json files are in root directory of your repos. You may save some time by using these two Rest APIs:
List all repos in current organization:
GET https://dev.azure.com/{OrganizationName}/_apis/git/repositories?api-version=5.1
Get File(Download):
GET https://dev.azure.com/{OrganizationName}/_apis/git/repositories/{Repositoryid}/items?scopePath=/appsettings.json&download=true&api-version=5.1
You can use PowerShell script to combine these two apis. The first one will list all Repositoryids in your organization, and the second one can download the appsettings.json from different repos via different Repositoryids. So the possible way could be run the first api once to get list of reposID(You can check this similar one) and then add a loop to get the files one by one.
I have got a project in an old org (from VSTS), that I want to move to my new one.
I can't see any options in Azure DevOps on migrating projects, or any information on the interwebs.
Anyone know how to do it?
If you just need to move repos, you can use the built in clone functions:
Go to the Azure Devops source repo -> Files
Click "Clone"
Choose "Generate Git Credentials"
Create the target repo in the target Azure DevOps
Choose "Import a repository"
Use the URL and credentials from Step 3
Done
This is not supported today. But this feature was planned to develop: make it possible to move a Team Project between Team Project Collections
If your Azure Devops project only tracks code versions using a single Git repo, hence no boards, user stories, tasks, pipelines, etc. then you can do the following:
Clone your project repo.
For example with Visual Studio.
You don't need to clone if you already have a local repo.
Destroy the association with the remote.
For this typically, you need to open a command line prompt in the folder that contains the .git database folder, most likely the solution folder of Visual Studio and type git remote rm origin.
Here is an example using git bash showing the content of the solution folder, including the .git database and the *.sln Visual Studio solution file:
Open the solution with Visual Studio if not already done.
It should now show that you have many commits waiting to be pushed to a remote. For illustration purpose, my toy project only have 8 commits in total.
Click the up arrow and choose your new remote, say a brand new Azure DevOps project, in the organization of your choice, then push.
You are now done cloning the project in another organization. If needed, then destroy the project in the old organization to complete the "move" operation.
There are 3 projects that I know of to achieve this.
A paid for option by Ops Hub -
OpsHub Visual Studio Migration Utility
An open source tool that requires making changes to the work item process template - Azure DevOps Migration tools
An lastly an Unofficial but still written by Microsoft tool to create Azure DevOps project templates - Azure DevOps Demo Generator & extractor tool
With the last one (the Demo Generator) you extract the project as a template, then apply it to the new organisation. As it is a tool for demo's there is no support provided and in my experience it works for simple projects but falls over on anything complex.
Expanding on others' answers, this post regards Pipelines.
Azure DevOps API
Migrating nearly all aspects of a project across organizations is doable, but it is a lot of manual work using the Azure DevOps API. The link below shows you all the end points, variables, etc. From there you'll probably want to write a Power Shell script and do a couple test runs to a dummy Organization.
https://learn.microsoft.com/en-us/rest/api/azure/devops/?view=azure-devops-rest-6.1&viewFallbackFrom=azure-devops-rest-6.0
In App options
If you avoid the API, there is no way to migrate pipelines that preserve build or release history, but you can preserve your configurations and processes by going into your Pipelines and selecting View YAML. From here you can either take this away as notes to recreate the GUI steps in your new org/project location, or actually adopt the YAML standard in your git repository.
I do not believe there is a way to migrate pipeline variables outside of the API. However, you can move the variables to Azure Key Vault and change your pipeline settings (YAML) to reference values from key vault. This is not a large amount of effort and is a nice process improvement.
Lastly, if you have any locally installed pipeline agents for releases, you will need to run the Power Shell script for your new organization on the boxes. Very simple 5m step, but right now the Agent Pools are not sharable across organization.
As #Frederic mentioned in his answer, we can actually easily do it with Visual Studio. I have done this without Visual Studio. The steps involved are below.
Add a User to Both Organization
Configure SSH Key
Update the SSH Key in Source DevOps and Clone the Repository
Check out all the Branches and Tags
Update the SSH Key in the Destination DevOps
Remove Old and Add New Origin
Push all the branches
The commands and detailed explanations can be found here.
BTW, if you need to change the entire Devops Organization tied to your personal Tenant (E.g. VS Enterprise Subscription) and move it to new Tenant, you can change the AAD and point it to the new one e.g. your EA Tenant on Azure commercial cloud.
Before you switch your organization directory, make sure the following statements are true:
You're in the Project Collection Administrator group for the
organization.
You're a member or a guest in the source Azure AD and a
member in the destination Azure AD
You have 100 or fewer users in
your source organization. Otherwise you will have to open a support ticket.
You may have to add the users back in destination org if they do not exist becuase they will loose access the moment you switch the AAD.
you could just download as a zip file and then download it to the destination repo