I have a self hosted agent and have a git repository with submodules. URLs in .gitmodules are http://
When I try to initialize a job it fails to update submodules.
git submodule sync
git submodule update --init --force
Cloning into 'foo-dev-common'...
Submodule 'foo-dev-common' (https://MY_ORG#dev.azure.com/MY_ORG/PInC/_git/foo-dev-common) registered for path 'foo-dev-common'
fatal: could not read Password for 'https://MY_ORG#dev.azure.com': terminal prompts disabled
fatal: clone of 'https://MY_ORG#dev.azure.com/MY_ORG/PInC/_git/foo-dev-common' into submodule path 'foo-dev-common' failed
##[error]Git submodule update failed with exit code: 128
Finishing: Checkout foo-rose-identity-service#submod_bd_mahesh to s/foo-rose-identity-service
I have also tried adding repository self and
steps:
- checkout: self
submodules: true
persistCredentials: true
forvaidya's answer didn't work for me (though it is now 4 years later).
(Relative URLs in .gitmodules are resolved to full URLs in .git/config by git submodule sync.)
persistCredentials: true will keep the authorization header available in git config for future steps, but it is keyed by your main repo URL. As long as the submodule repo(s) are in the same organization, you can reuse the header, though - e.g. in a pipeline Powershell script:
steps:
- checkout: self
submodules: false
persistCredentials : true
- powershell: |
$header = $(git config --get-all http.$(Build.Repository.Uri).extraheader)
git -c http.extraheader="$header" submodule sync
git -c http.extraheader="$header" submodule update --init --force --depth=1
(I gleaned these details from the logs of the standard checkout step. Note the reference to the Build.Repository.Uri pipeline variable.)
The above will accomplish a full ("unshallow") checkout of the main repo (useful for e.g. GitVersion), without submodules, and a shallow checkout of any submodules.
Edit: The documented way to get the authorization header is
$header = "AUTHORIZATION: bearer $(System.AccessToken)"
After making git submodule relative path it worked
url= ../foo-dev-common
instead of
url=https://MY_ORG#dev.azure.com/MY_ORG/PInC/_git/foo-dev-common
Related
I have a very simple Yaml code, I am trying to use and it is causing me so much pain but I cannot see an error besides the one i am getting at the end
error: pathspec 'main' did not match any file(s) known to git
[detached HEAD d1f4c36] Work now
Here is the code :
stages:
- stage : Build
jobs:
- job:
displayName: "Build Ripple 3 dataverse Solution"
pool :
vmImage: 'windows-2019'
variables:
- group: "Ripple 3 Core"
steps:
- task: PowerPlatformToolInstaller#2
inputs:
DefaultVersion: true
displayName : "Power Platform Tool Installer"
- task: PowerPlatformExportSolution#2
inputs:
authenticationType: 'PowerPlatformSPN'
PowerPlatformSPN: 'Service Connection'
SolutionName: '$(PowerPlatformSolution)'
SolutionOutputFile: '$(Pipeline.Workspace)\$(PowerPlatformSolution).zip'
- task: PowerPlatformUnpackSolution#2
inputs:
SolutionInputFile: '$(Pipeline.Workspace)\Ripple3Core.zip'
SolutionTargetFolder: '$(Build.SourcesDirectory)\$(PowerPlatformSolution)_unmanaged'
- script: |
git config user.email "xx#xx.com"
git config user.name "Automatic Build"
git checkout main
git add --all
git commit -m "Work now"
git push origin main
Try
git checkout "main"
Also, it may be necessary to checkout before exporting solution :
https://zupimages.net/viewer.php?id=22/40/z330.png
It is also possible to use Publish artifacts component :
https://learn.microsoft.com/en-us/azure/devops/pipelines/artifacts/pipeline-artifacts?view=azure-devops&tabs=yaml
Git is telling you that it does not recognize 'main' as branch nor file. The branches only exist on remote repository unless you checked them out already.
Instead of
git checkout main # there is no local branch called main
Try
git checkout origin/main
Try adding git fetch before your checkout.
git fetch
git checkout main
I am trying to setup auto merge for my main branch whenever something is pushed into the development branch using gitaction. This is my gitaction file:
name: Auto merge
on:
push:
branches:
- development
env:
# replace "github_username" with your GitHub username
# replace "github.com/username/repo.git" with your GitHub repo path
# do NOT replace ${{secrets.GITHUB_TOKEN}}, GitHub will take care of it
MY_REPO: https://my-username:${{secrets.GITHUB_TOKEN}}#github.com/organisation-username/repo.git
# replace "long-lived_branch_name" with your branch name
MY_BRANCH: development
# replace it with the path to master repo
MAIN_REPO: https://github.com/organisation-username/repo.git
# replace "master" with your master branch name
MAIN_BRANCH: main
jobs:
merge:
runs-on: ubuntu-latest
steps:
- name: Merge with master
run: |
git clone ${{env.MY_REPO}} -b ${{env.MY_BRANCH}} tmp
cd tmp
git config user.name "Automerge Bot"
git config user.email "gcpabia#gmail.com"
git config pull.rebase false
git pull ${{env.MAIN_REPO}} ${{env.MAIN_BRANCH}}
git push
The thing is I belong to the organization as owner, so the repo isn't in my personal account. I keep getting this error:
Run git clone ***github.com/organization-username/repo.git -b development tmp
Cloning into 'tmp'...
fatal: could not read Username for 'https://github.com': No such device or address
Error: Process completed with exit code 1.
My suspicion is that the issues might be the different usernames I am using to carryout this operation and have switched between that of the organization and my personal account with no success.
Any pointers as to what I am missing.
My repository in my organisation's devops project contains a lot of .net solutions and some unity projects as well. When I run my build pipeline, it fails due to several of these:
Error MSB3491: Could not write lines to file "obj\Release\path\to\file". There is not enough space on the disk.
I would like the pipeline to only checkout and fetch parts of the repository that are required for a successful build. This might also help with execution time of the pipeline since it currently also fetches the whole of my unity projects with gigabytes of resources which takes forever.
I would like to spread my projects across multiple repositories but the admin won't give me more than the one I already have. It got a lot better when I configured git fetch as shallow (--depth=1) but I still get the error every now and then.
This is how I configured the checkout:
steps:
- checkout: self
clean: true
# shallow fetch
fetchDepth: 1
lfs: false
submodules: false
The build is done using VSBuild#1 task.
I can't find a valid solution to my problem except for using multiple repositories, which is not an option right now.
Edit: Shayki Abramczyk's solution #1 works perfectly. Here is my full implementation.
GitSparseCheckout.yml:
parameters:
access: ''
repository: ''
sourcePath: ''
steps:
- checkout: none
- task: CmdLine#2
inputs:
script: |
ECHO ##[command] git init
git init
ECHO ##[command] git sparse-checkout: ${{ parameters.sourcePath }}
git config core.sparsecheckout true
echo ${{ parameters.sourcePath }} >> .git/info/sparse-checkout
ECHO ##[command] git remote add origin https://${{ parameters.repository }}
git remote add origin https://${{ parameters.access }}#${{ parameters.repository }}
ECHO ##[command] git fetch --progress --verbose --depth=1 origin master
git fetch --progress --verbose --depth=1 origin master
ECHO ##[command] git pull --progress --verbose origin master
git pull --progress --verbose origin master
Checkout is called like this (where template path has to be adjusted):
- template: ../steps/GitSparseCheckout.yml
parameters:
access: anything:<YOUR_PERSONAL_ACCESS_TOKEN>
repository: dev.azure.com/organisation/project/_git/repository
sourcePath: path/to/files/
In Azure DevOps you don't have option to get only part of the repository, but there is a workaround:
Disable the "Get sources" step and get only the source you want by manually executing the according git commands in a script.
To disable the default "Get Sources" just specify none in the checkout statement:
- checkout: none
In the pipeline add a CMD/PowerShell task to get the sources manually with one of the following 2 options:
1. Get only part of the repo with git sparse-checkout.
For example, get only the directories src_1 and src_2 within the test folder (lines starting with REM ### are just the usual batch comments):
- script: |
REM ### this will create a 'root' directory for your repo and cd into it
mkdir myRepo
cd myRepo
REM ### initialize Git in the current directory
git init
REM ### set Git sparsecheckout to TRUE
git config core.sparsecheckout true
REM ### write the directories that you want to pull to the .git/info/sparse-checkout file (without the root directory)
REM ### you can add multiple directories with multiple lines
echo test/src_1/ >> .git/info/sparse-checkout
echo test/src_2/ >> .git/info/sparse-checkout
REM ### fetch the remote repo using your access token
git remote add -f origin https://your.access.token#path.to.your/repo
REM ### pull the files from the source branch of this build, using the build-in Azure DevOps variable for the branch name
git pull origin $(Build.SourceBranch)
displayName: 'Get only test/src_1 & test/src_2 directories instead of entire repository'
Now in the builds task make myRepo the working directory.
Fetching the remote repo using an access token is necessary, since using checkout: none will prevent your login credentials from being used.
In the end of the pipeline you may want to add step to clean the myRepo directory.
2. Get parts of the repo with Azure DevOps Rest API (Git - Items - Get Items Batch).
The other answers work well but I found a different way using potentially newer features of git.
This will fetch to a depth of 1 and show all the files in the root folder plus folder1, folder2 and folder3
- task: CmdLine#2
inputs:
script: |
git init
git sparse-checkout init --cone
git sparse-checkout set folder1 folder2 folder3
git remote add origin https://<github-username>:%GITHUB_TOKEN%#<your-git-repo>
git fetch --progress --verbose --depth=1 origin
git switch develop
env:
GITHUB_TOKEN: $(GITHUB_TOKEN)
Maybe it is helpful for you to check out only a specific branch. This works by:
resources:
repositories:
- repository: MyGitHubRepo
type: github
endpoint: MyGitHubServiceConnection
name: MyGitHubOrgOrUser/MyGitHubRepo
ref: features/tools
steps:
- checkout: MyGitHubRepo
Or by using the inline syntax like so
- checkout: git://MyProject/MyRepo#features/tools # checks out the features/tools branch
- checkout: git://MyProject/MyRepo#refs/heads/features/tools # also checks out the features/tools branch
- checkout: git://MyProject/MyRepo#refs/tags/MyTag # checks out the commit referenced by MyTag.
More information can be found here
A Solution For Pull Request and Master Support
I realized after posting this solution it is similar to the updated one on the post. However this solution is a bit more rich and optimized. But most importantly this solution uses the pull request merge branch in Dev Ops for the deployments like the native checkouts do. It also fetches only the needed commits.
Supports multiple folder/path patterns as parameters
Minimal checkout with the bare minimum needed via sparse checkout
Shallow depth, multithreaded fetch, with a sparse index.
It takes into account using the PR merge branch against main rather than the raw PR branch itself if needed.
Uses native System Token already in pipeline
Handles detection and alternative ref flows for master where a merge branch does not exist.
Example Use in your Script:
- job: JobNameHere
displayName: JobDisplayName Here
steps:
- template: templates/sparse-checkout.yml
parameters:
checkoutFolders:
- /Scripts
- /example-file.ps1
# other steps
templates/sparse-checkout.yaml
parameters:
- name: checkoutFolders
default: '*'
type: object
steps:
- checkout: none
- task: PowerShell#2
inputs:
targetType: inline
script: |
$useMasterMergeIfAvaiable = $true
$checkoutFolders = ($env:CheckoutFolders | ConvertFrom-Json)
Write-Host $checkoutFolders
$sw = [Diagnostics.Stopwatch]::StartNew() # For timing the run.
$checkoutLocation = $env:Repository_Path
################ Setup Variables ###############
$accessToken = "$env:System_AccessToken";
$repoUriSegments = $env:Build_Repository_Uri.Split("#");
$repository = "$($repoUriSegments[0]):$accessToken#$($repoUriSegments[1])"
$checkoutBranchName = $env:Build_SourceBranch;
$prId = $env:System_PullRequest_PullRequestId;
$repositoryPathForDisplay = $repository.Replace("$accessToken", "****");
$isPullRequest = $env:Build_Reason -eq "PullRequest";
################ Configure Refs ##############
if ($isPullRequest)
{
Write-Host "Detected Pull Request"
$pullRequestRefMap = "refs/heads/$($checkoutBranchName):refs/remotes/origin/pull/$prId"
$mergeRefMap = "refs/pull/$prId/merge:refs/remotes/origin/pull/$prId";
$mergeRefRemote = $mergeRefMap.Split(":")[0];
$remoteMergeBranch = git ls-remote $repository "$mergeRefRemote" # See if remote merge ref exiss for PR.
if ($useMasterMergeIfAvaiable -and $remoteMergeBranch)
{
Write-Host "Remote Merge Branch Found: $remoteMergeBranch" -ForegroundColor Green
$refMapForCheckout = $mergeRefMap
$remoteRefForCheckout = "pull/$prId/merge"
}else{
Write-Host "No merge from master found (or merge flag is off in script), using pullrequest branch." -ForegroundColor Yellow
$refMapForCheckout = $pullRequestRefMap
$remoteRefForCheckout = "heads/$checkoutBranchName"
}
$localRef = "origin/pull/$prId"
}else{
Write-Host "This is not a pull request. Assuming master branch as source."
$localRef = "origin/master"
$remoteRefForCheckout = "master"
}
######## Sparse Checkout ###########
Write-Host "Beginning Sparse Checkout..." -ForegroundColor Green;
Write-Host " | Repository: $repositoryPathForDisplay" -ForegroundColor Cyan
if (-not (test-path $checkoutLocation) ) {
$out = mkdir -Force $checkoutLocation
}
$out = Set-Location $checkoutLocation
git init -q
git config core.sparsecheckout true
git config advice.detachedHead false
git config index.sparse true
git remote add origin $repository
git config remote.origin.fetch $refMapForCheckout
git sparse-checkout set --sparse-index $checkoutFolders
Write-Host " | Remote origin configured. Fetching..."
git fetch -j 4 --depth 1 --no-tags -q origin $remoteRefForCheckout
Write-Host " | Checking out..."
git checkout $localRef -q
Get-ChildItem -Name
# tree . # Shows a graphical structure - can be large with lots of files.
############ Clean up ##################
if (Test-Path -Path ..\$checkoutLocation)
{
Write-Host "`nChecked Out`n#############"
Set-Location ../
}
$sw.Stop()
Write-Host "`nCheckout Complete in $($sw.Elapsed.TotalSeconds) seconds." -ForegroundColor Green
displayName: 'Sparse Checkout'
env:
Build_Repository_Uri: $(Build.Repository.Uri)
Build_Reason: $(Build.Reason)
System_PullRequest_SourceBranch: $(System.PullRequest.SourceBranch)
System_PullRequest_PullRequestId: $(System.PullRequest.PullRequestId)
System_PullRequest_SourceRepositoryURI: $(System.PullRequest.SourceRepositoryURI)
Build_BuildId: $(Build.BuildId)
Build_SourceBranch: $(Build.SourceBranch)
CheckoutFolders: ${{ convertToJson(parameters.checkoutFolders) }}
System_AccessToken: $(System.AccessToken)
Repository_Path: $(Build.Repository.LocalPath)
With LFS support on Ubuntu and Windows agents
parameters:
folders: '*'
steps:
- bash: |
set -ex
export ORIGIN=$(Build.Repository.Uri)
export REF=$(Build.SourceVersion)
export FOLDERS='${{ parameters.folders }}'
git version
git lfs version
git init
git sparse-checkout init --cone
git sparse-checkout add $FOLDERS
git remote add origin $ORIGIN
git config core.sparsecheckout true
git config gc.auto 0
git config advice.detachedHead false
git config http.version HTTP/1.1
git lfs install --local
git config uploadpack.allowReachableSHA1InWant true
git config http.extraheader "AUTHORIZATION: bearer $(System.AccessToken)"
git fetch --force --no-tags --progress --depth 1 origin develop $REF
git checkout $REF --progress --force
displayName: Fast sparse Checkout
Then use as a step
steps:
- checkout: none
- template: fastCheckout.yaml
parameters:
folders: 'Folder1 src/Folder2'
You can pass folders as paramters
The exports are there to make it easier to test the script locally.
Improved checkouts from 10mins to 2mins
I am building an Azure pipeline for a GitHub web site to run on Windows Self-Hosted agents.
The default branch for the GitHub project is develop, all developers commit to this branch. I want a script that will merge develop to release for the time where a version is on tests and merge release to master once in production.
I'm new with git commands, I know the pipeline runs under a service account on the agent, behind a proxy and the pipeline impersonate somehow to another account to connect to GitHub.
To test my script, I logged onto the server as the service account and ran the following commands:
REM At start, the pipeline is on the develop branch so I move to release branch
C:\Agent\_work\29\s> git checkout release
>Updating files: 100% (928/928), done.
>Previous HEAD position was a62***: *comment*
>Switched to a new branch 'release'
>Branch 'release' set up to track remote branch 'release' from 'origin'.
C:\Agent\_work\29\s>git tag "branchTests"
C:\Agent\_work\29\s>git status
>On branch release
>Your branch is up to date with 'origin/release'.
>
>nothing to commit, working tree clean
REM I understood I had to first get the release branch and then pull the develop branch over it before pushing it all back
C:\Agent\_work\29\s>git pull origin develop
fatal: could not read Username for 'https://github.com': No such file or directory
C:\Agent\_work\29\s>git push --verbose --repo=release --set-upstream release
>Pushing to release
>fatal: 'release' does not appear to be a git repository
>fatal: Could not read from remote repository.
>
>Please make sure you have the correct access rights and the repository exists.
I have two questions:
Is my script correct in that context ?
Can this username error come from that I use the service account ? I should impersonate somehow as the same account the pipeline does ?
Thanks.
########## Update 1
I noticed that I don't need to add this if I don't disable the checkout in the pipeline
git config --global user.email "you#example.com"
git config --global user.name "xxx"
git remote set-url origin https://user:{GitHubPAT}#github.com/xxx/xxx.git
I have two files:
stages:
- stage: InitRelease
jobs:
- job: Branch
steps:
- checkout: self
clean: true
persistCredentials: true
- template: git-branch-source-2-target.yml#templates
parameters:
Tag: '${{ variables.projectName }}_${{ variables.buildId }}'
git-branch-source-2-target.yml
parameters:
- name: 'SourceBranch'
default: 'develop'
type: string
- name: 'TargetBranch'
default: 'release'
type: string
- name: 'Tag'
default: ''
type: string
steps:
- task: CmdLine#2
enabled: true
displayName: 'GIT Release'
inputs:
script: |
git checkout ${{parameters.SourceBranch}}
git pull origin
git checkout ${{parameters.TargetBranch}}
git pull origin
git tag ${{parameters.Tag}}
git push --tags
git merge ${{parameters.SourceBranch}}
git push origin --all --verbose
Thanks !
Run the following command in the cmd task of the pipeline:
git config --global user.email "you#example.com"
git config --global user.name "xxx"
git remote set-url origin https://user:{GitHubPAT}#github.com/xxx/xxx.git
git checkout develop
git pull origin
git checkout release
git merge develop
git push origin --all
I want to publish releases to gh-pages. The deploy provider can be configured to keep history using keep-history: true. However I would like previous versions not to be just available in the git history but not to be deleted from the repository.
I've configured yarn and webpack to create a separate directory for each tag and to put the distribution in both a "latest"-directory as well as this tag specific directory. I would like to see a tag directory for all previous versions and not just for the latest version.
Here are the results of my current configuration: https://github.com/retog/rdfgraphnode-rdfext/tree/gh-pages
I found the following solution.
In travis.yml replace:
- provider: pages
skip-cleanup: true
github-token: $GITHUB_TOKEN
keep-history: true
local-dir: distribution
on:
tags: true
With:
- provider: script
script: bash .travis_publish
on:
tags: true
And add the script file .travis_publish with the following content:
#!/bin/bash
PUBLICATION_BRANCH=gh-pages
# Checkout the branch
REPO_PATH=$PWD
pushd $HOME
git clone --branch=$PUBLICATION_BRANCH https://${GITHUB_TOKEN}#github.com/$TRAVIS_REPO_SLUG publish 2>&1 > /dev/null
cd publish
# Update pages
cp -r $REPO_PATH/distribution .
# Commit and push latest version
git add .
git config user.name "Travis"
git config user.email "travis#travis-ci.org"
git commit -m "Updated distribution."
git push -fq origin $PUBLICATION_BRANCH 2>&1 > /dev/null
popd