I am writing an Azure PowerShell script that will consume the JSON file which has the location of all my SQL scripts and Migrationflag column (which holds to execute/to be executed) and execute all the sequence of scripts.
upon execution, the flag will change to 'N' and the updated JSON file should be uploaded to bitbucket.
Now, I am stuck with "fatal: not a git repository (or any of the parent directories): .git" error while trying to push.
I've created a pat token and service connection with username: santhoshsreshta and below is the code to push.
$v_JSON = Get-Content '$(system.defaultworkingdirectory)\locationToBuild\BuildOrder.json' -Raw | ConvertFrom-Json
$v_JSON | Sort-Object -Property OrderNo | Where-Object {$_.MigratedFlag -like 'Y'} | ForEach {
$Script = $_.Location
Write-Host "Executing Script: $Script"
Invoke-Sqlcmd -ServerInstance "myservername" -Database $(database) -Username $(testauto_username) -Password $(testauto_password) -InputFile $(system.defaultworkingdirectory)\$Script
$_.MigratedFlag = 'N'
}
$v_JSON | ConvertTo-Json -depth 32| set-content '$(system.defaultworkingdirectory)\locationToBuild\BuildOrder.json'
$MyPat = 'mypatcode'
git push https://mygitusername:$MyPat#bitbucket.org/xyz/abcd.git
getting the error,"##[error]fatal: Not a git repository (or any of the parent directories): .git"
but when issuing git clone https://mygitusername:$MyPat#bitbucket.org/xyz/abcd.git -- getting invalid username/password error.
I believe we should not clone again as my pipelines get sources task will clone it and puts in a self-hosted agent.
this is my git url: https://mygitusername#bitbucket.org/xyz/abcd.git
Thanks a ton,
A DevOps, PowerShell newbie here.
Receiving “fatal: not a git repository (or any of the parent directories): .git” while using pat to push code into bitbucket using azure powershell
That because we are using git feature, but we are not in the folder managed by git.
When we use Azure powershell task directly, the default work folder should be:
PS C:\Users\<Username>
Obviously, there are no files managed by our git in this directory. That the reason why you get the error Not a git repository.
So, to resolve this issue, we just need to switch the working folder to the repo folder by the command:
cd $(System.DefaultWorkingDirectory)
Check the Use predefined variables and this thread for some details.
Update:
The test sample:
cd $(System.DefaultWorkingDirectory)
cd leotest
echo 123>Test.txt
git add Test.txt
git commit -m "Add a test file"
git push https://Username:password#bitbucket.org/Lsgqazwsx/leotest.git HEAD:master
It means that there is no local .git & you need to do the below first:
git init
git commit -m "first commit"
git branch -M main
git remote add origin https://your-repo-url
git push -u origin main
I just copied the below from GitHub page & it works for Azure Repos :-) further showing the power & advantage of companies following universal standards
I'm looking for a way to create an artifact that I can attach to a deployment pipeline that only contains the files that were changed in the commits that triggered this build.
What I have is a repo that has change scripts for database objects, so I want to package up only the change scripts for the last commit into a zip file and attach it to the build outputs. That way I can take zip file and apply each of the files on top of the database, this will be done later in a different step, right now I'm just trying to get all of the files that were changed.
Editted
I have created the following step in the YAML file based on the comments below
- powershell: |
#get the changed template
echo "git diff-tree --no-commit-id --name-only -r $(Build.SourceVersion)"
$a = git diff-tree --no-commit-id --name-only -r $(Build.SourceVersion)
#assign the filename to a variable
echo "Files"
echo "##vso[task.setvariable variable=fileName]$a"
- powershell: |
#Print Files
$fileName: echo "$env:fileName"
Below is the result, you can see that no files are changed. Here I changed the Readme file, which triggered the build.
Not sure if this would help you, but hopefully will point in the right direction.
Assuming you have Git as source control. Have you considered to query those changes using Git instead? (I haven't tried, but I bet you'll be able to find in the pipeline metadata which merge triggered the build, and then use it to query Git for the file changes.
Have a look at this question in Stackoverflow
Hope it will help.
If you are using Git version control, you could try to add a script task to get the changed file names in your pipeline, copy them to artifact directory and then publish them.
It is easy to get the changed files using git commands git diff-tree --no-commit-id --name-only -r commitId. When you get the changed file's name, you need to assign it to a variable using expression ##vso[task.setvariable variable=VariableName]value. Then you can use this variable in the copy and publish task.
You can check below yaml pipeline for example:
- powershell: |
#get the changed template
$a = git diff-tree --no-commit-id --name-only -r $(Build.SourceVersion)
#assign the filename to a variable
echo "##vso[task.setvariable variable=fileName]$a"
- powershell: |
echo "$env:fileName"
I have created powershell script to update json file with variables. Json file is located in Azure devops repo, json file name var.json.
I am going to use this solution in azure devops, so I built pipeline and set test variable in variables tab in azure devops:
In my script I have param and variables blocks, presented below:
param(
[Parameter (Mandatory=$true)]
[String] $FileRes
)
#env variable
$Path = $Env:BUILD_SOURCESDIRECTORY
# Download variables from Json file
$JsonBase = #()
$JsonPath = "$Path\Var.json"
$JsonBase = Get-Content $JsonPath | out-string | ConvertFrom-Json
$JsonBase.FileNames[0].value = $FileRes
in my script I use commands: $JsonBase | ConvertTo-Json | Set-Content -Path $JsonPath to direct output to json file.
Json file structure:
{
"FileNames": [
{
"value": "AAAbbbccc123",
"value1": "www",
"value3": "swd",
"value4": "xvb"
}
]
}
Pipeline's status at the end is ok, all steps are green, but var.json file is not updated as I wanted. There is still old value --> "value": "AAAbbbccc123"
In fact, it has been replaced, but you need to see this change in the output repos.
For more clearly, you could use private agent to run this build. Then go the corresponding local repos and check the Var.json file after the build finished:
In your script, you are Set-Content into the file which exists under the $(Build.SourcesDirectory)\Var.json, not the one which stored in VSTS repos. So, to check whether it is replaced successfully, please go your output repos, the one in agent.
Sometimes, if what you used is hosted agent, you may could not view the detailed output repos since the host image will be recycled by the server after the pipeline finished.
At this time, you can add another script in it to print the JSON file content out, then you could check whether it is replaced successfully:
$content= Get-Content -Path $JsonPath
Write-Host $content
In addition, please make a little change into your script:
$JsonBase.FileNames[0].value = "$(FileRes)"
Here please use $(FileRes) instead of $FileRes, since you specified the value in the Variables tab. And do not forget the double quote "".
Update:
To sync the output repos change back into VSTS repos, try follow:
(1) The first command line task:
git config --global user.email "xxx#xx.com"
git config --global user.name "Merlin"
cd $(Build.SourcesDirectory)
git init
(2) In powershell task, execute set-content script.
(3) In second command line task, do git push to push the changes:
git add Var.json
git commit -m "aaaa"
git remote rm origin
git remote add origin https://xxx#dev.azure.com/xxx/xxx/_git/xxxx
git push -u origin HEAD:master
In addition, to run git script successfully in pipeline. Beside enable “Allow script to access........” you also should follow this permission setting.
Is it somehow possible to get the pull request number in a Visual Studio Team Service (vNext) build which is linked to a GitHub repository for builds run for pull requests?
I would like to do some code anylsis using sonar and write the finding back as comment to the pull request using Sonar GitHub Plugin.
I don't know any direct way to do this. The way I can think is add a PowerShell step to call "git log" command and read the information from the log. Since the commit information for pull request usually has a format like "Merge pull request #6 from XXX". We can use RegEx to get the pull request number.
git log -1 >log.txt
$file = Get-Content log.txt
$reg = "Merge.pull.request.+(?<pullnumber>\w+?).from+"
foreach($line in $file){
if($line -match $reg){
$Matches.pullnumber;
}
}
Although this question is similar to GitHub latest release, it's actually different -- it's about a link that means "the latest version of the download file itself".
GitHub provides a "Latest" URL that redirects to the information page for the latest release. For example: https://github.com/reactiveui/ReactiveUI/releases/latest will redirect to https://github.com/reactiveui/ReactiveUI/releases/tag/5.99.6 (as I type this; or to the page for a newer version, someday).
That's great but I need a URL to the download file itself. In this example, the .zip file associated with the green download button, https://github.com/reactiveui/ReactiveUI/releases/download/5.99.6/ReactiveUI-5.99.6.zip (as I type this; or to a newer zip file, someday).
Why? I want to give the URL to curl, as part of a Travis CI script, to download the latest version.
I guessed at a few URLs like /releases/download/latest/file.zip (substituting "latest" for the version part) and /releases/download/file.zip but those 404.
Is there any way to do this -- in the context of a shell script and curl (note: not in a browser page with JS)?
For releases that do not contain the version number or other variable content in their assets' names, you can use a URL of the format:
https://github.com/owner/repository/releases/latest/download/ASSET.ext
As per the docs:
If you'd like to link directly to a download of your latest release asset you can link to /owner/name/releases/latest/download/asset-name.zip.
Here is a way to do it w/o Github if you have a single download in the release:
wget $(curl -s https://api.github.com/repos/USERNAME/REPONAME/releases/latest | grep 'browser_' | cut -d\" -f4)
It is pretty easy (though not pretty), and of course you can swap out wget for another curl call if you want to pipe it to something.
Basically, the curl call nets you a JSON structure, and I'm just using basic shell utilities to extract the URL to the download.
Very interesting, I haven't noticed a "latest" tag in GitHub-releases yet. As i now figured out, they're given away if you're using the "pre-release"-capabilities of GitHubs release-system. But i don't know any way to access binaries via a latest-path.
I would like to suggest you using git (which is available in your travis-vm) to download the latest tag.
Like Julien Renault describes in his blog post, you will be able to checkout the latest tag in the repository like this:
# this step should be optional
git fetch --tags
latestTag=$(git describe --tags `git rev-list --tags --max-count=1`)
git checkout $latestTag
This solution is based on the assumption that the latest tag is also the latest version.
I use this to get the download URLs in PowerShell 5+ (replace ACCOUNT & REPO)
Invoke-RestMethod -uri https://api.github.com/repos/ACCOUNT/REPO/releases/latest | select -ExpandProperty assets | select -expand browser_download_url
Note if they have more than one package this will be a list. If you want to pick a certain one find a unique part of the name i.e. win for Windows and use:
(replace ACCOUNT, REPO & SELECTOR)
Invoke-RestMethod -uri https://api.github.com/repos/ACCOUNT/REPO/releases/latest | select -ExpandProperty assets | ? { $_.name.Contains("SELECTOR")} | select -expand browser_download_url
As a bonus if you assign the above to a variable you can then grab the file and extract it with the following (assuming you assign to $uri):
Invoke-WebRequest $uri -OutFile "release.zip"
Expand-Archive .\release.zip
In PowerShell 6+ this should work on other platforms than Windows.
On windows, only using powershell, this works for me. It can probably be written a lot shorter.
#Downloads latest paket.bootstrapper.exe from github
$urlbase = "https://github.com"
$latestPage="$urlbase/fsprojects/Paket/releases/latest"
Write-Host "Parsing latest release page: $latestPage"
$page=Invoke-Webrequest -uri $latestPage
$latestBootStrapper=($page.Links | Where-Object { $_.href -match "bootstrapper" }).href
$dlurl="$urlbase$latestBootStrapper"
Write-Host "Downloading paket.bootstrapper.exe from $dlurl"
$wc=new-object net.webclient
$wc.UseDefaultCredentials=$true
$wc.Proxy.Credentials=$wc.Credentials
$wc.DownloadFile($dlurl, (join-path (resolve-path ".\") "paket.bootstrapper.exe"))
$repoName = "PowerShell/PowerShell"
$assetPattern = "*-win-x64.msi"
$extractDirectory = "C:\Users\Public\Downloads"
$releasesUri = "https://api.github.com/repos/$repoName/releases/latest"
$asset = (Invoke-WebRequest $releasesUri | ConvertFrom-Json).assets | Where-Object name -like $assetPattern
$downloadUri = $asset.browser_download_url
$extractPath = [System.IO.Path]::Combine($extractDirectory, $asset.name)
Invoke-WebRequest -Uri $downloadUri -Out $extractPath
You can use curl with https://api.github.com. It gives JSON output from which you can easily extract what you need with jq or your favorite json tool.
For example, using the repository in the question:
gituser=reactiveui; repo=ReactiveUI
tag_name=$(curl -sL https://api.github.com/repos/$gituser/$repo/releases/latest | jq -r '.tag_name'); echo $tag_name
# output: "16.3.10"
tarurl=$(curl -sL https://api.github.com/repos/$gituser/$repo/releases/latest | jq -r '.tarball_url'); echo $tarurl
# output: https://api.github.com/repos/reactiveui/ReactiveUI/tarball/16.3.10
zipurl=$(curl -sL https://api.github.com/repos/$gituser/$repo/releases/latest | jq -r '.zipball_url'); echo $zipurl
# output: https://api.github.com/repos/reactiveui/ReactiveUI/zipball/16.3.10
So you could get the download with a nested curl in a one-liner:
curl -OL $(curl -sL https://api.github.com/repos/filesender/filesender/releases/latest | jq -r '.tarball_url')
This will download the file, and save it with the name of its tag_name, but without extension. So you may want to rename it by appending ".tgz" or ".zip", depending on which you downloaded.
Note for Windows users: curl is now installed by default on Windows too, but beware that it must be called as curl.exe. That's because Powershell has an alias stupidly called "curl" which is not the same!
Centos/RHEL
There are 2 options to download using the URL directly.
Via Github API (using CURL and jq package)
Via Github direct (using CURL and sed)
I am listing a demonstration script for each option.
Option 1
#!/bin/bash
# author: fullarray
# Contribution shared on: stackoverflow
# Contribution shared on: github
# date: 06112022
compose_version=$(curl https://api.github.com/repos/docker/compose/releases/latest | jq .name -r)
get_local_os_build=$(uname -s)-$(uname -m)
curl -L https://github.com/docker/compose/releases/download/$compose_version/docker-compose-$get_local_os_build
Option 2
#!/bin/bash
# author: fullarray
# Contribution shared on: stackoverflow
# Contribution shared on: github
# date: 06112022
get_local_os_build=$(uname -s)-$(uname -m)
compose_latest_version=$(curl -L "https://github.com/docker/compose/releases/download/`curl -fsSLI -o /dev/null -w %{url_effective} https://github.com/docker/compose/releases/latest | sed 's#.*tag/##g' && echo`/docker-compose-$get_local_os_build")
If you are fine with cloning the repository first, you can use git tag, which also allows you to sort the tags by version in various ways.
git clone https://github.com/reactiveui/ReactiveUI.git .
LATEST="$(git tag --sort=v:refname | tail -n1)"
git checkout "$LATEST"
This allows for more flexibility, as you can filter the tags you're not interested in with grep, e.g.:
git tag --sort=v:refname | grep -vE '-RC[0-9]+$' | tail -n1
Here's an excerpt from the documentation on git-tag:
Sort based on the key given. Prefix - to sort in descending order of the value. You may use the --sort=<key> option multiple times, in which case the last key becomes the primary key. Also supports version:refname or v:refname (tag names are treated as versions). The version:refname sort order can also be affected by the versionsort.suffix configuration variable. The keys supported are the same as those in git for-each-ref. Sort order defaults to the value configured for the tag.sort variable if it exists, or lexicographic order otherwise. See git-config(1).
If you really don't want to clone the repository, the --sort option also works with git ls-remote. It'll just take a bit more work to get the part you're interested in:
git ls-remote --tags --sort=v:refname https://github.com/reactiveui/ReactiveUI.git | awk -F'/' '{ print $NF }'
This approach doesn't seem to work all too well for the ReactiveUI repository in particular, because their tags are a bit messy, but it's an option.
Please note, that the sorting isn't quite the same as with semantic versioning, but git does allow you to work around most of these cases. As an example mqtt2prometheus has release candidates using the suffix RC1, RC2 etc., but git sorts 0.1.6-RC1 as being newer than 0.1.6. You can tell git that "RC" is a pre-release suffix to make it sort them correctly.
git tag -c 'versionsort.suffix=-RC' --sort=v:refname | tail -n1
Here's an excerpt from the documentation on git-config:
By specifying a single suffix in this variable, any tagname containing that suffix will appear before the corresponding main release. E.g. if the variable is set to "-rc", then all "1.0-rcX" tags will appear before "1.0". If specified multiple times, once per suffix, then the order of suffixes in the configuration will determine the sorting order of tagnames with those suffixes. E.g. if "-pre" appears before "-rc" in the configuration, then all "1.0-preX" tags will be listed before any "1.0-rcX" tags.
You can also sort by the date of the tag using --sort=taggerdate, that might work better in some situations.
As #florianb pointed out, I should use git.
Originally my .travis.yml was something like:
before_install:
- curl -L https://raw.githubusercontent.com/greghendershott/travis-racket/master/install-racket.sh | bash
This would automatically get whatever the latest version is, from the repo.
But someone pointed out to me that GitHub doesn't want people to use raw.github.com for downloads. Instead people should use "releases". So I was a good doob and manually made a release each time. Then my .travis.yml was something like:
before_install:
- curl -L https://github.com/greghendershott/travis-racket/releases/download/v0.6/install-racket.sh | bash
But it's a PITA to make a release each time. Worse, all .travis.yml files need to be updated to point to the newer version of the file.
Instead -- just use git to clone the repo, and use the file within it:
before_install:
- git clone https://github.com/greghendershott/travis-racket.git
- cat travis-racket/install-racket.sh | bash # pipe to bash not sh!