Accessing AppVeyor Project Environment Variables during manual deploy - appveyor

I'm new to AppVeyor, I work for a web agency where we have many projects in separate GIT repos.
Each project has a development branch which is what I'm watching in AppVeyor. Since we have a single internal development server running IIS, it was a good candidate to define our development server as an AppVeyor Environment. The development server is running the AppVeyor Agent.
Inside the project specific YAML files I'm specifying the environment name and a custom environment variable which I've defined.
environment:
iis_site_name: project-specific-site-name.com
deploy:
- provider: Environment
name: dev-environment
I've configured the AppVeyor environment to accept the environment variable from the project like this.
AppVeyor Environment - Note: websitebuild is the "deploy name" associated with the artifact.
This work's perfectly on a commit, the project is built and deployed to the Agent in the correct location.
Where this is not working is when I need to kick off a manual deploy. So lets say I want to kick off a manual deploy by going into the AppVeyor interface and selecting Environments > Dev Environment > New Deploy > Select Project
When this deploy runs the environment variable from the YAML file (iis_site_name) is not seen and a new IIS site named 'default' is created and the site is deployed there.
Note that I have also tried adding the environment variable in the project settings via the GUI (as opposed to in the YAML) and it behaves no differently.

I believe it is expected behavior, when build-time environment variables are not available, Appveyor will send to agent default value which you define at Environment level variables. Please look here
At the bottom of that screen we are defining its “default” value, i.e.
the value used when you deploy from Environments and build environment
variables are not present
As I understand this does not fit your scenario, because you need different site name for different projects. What you can do is to create separate Environment for every project (with separate default site names), but replace all their environment access keys with the same you use for you current environment, thus agent will pick up jobs from all of them.
Update:
This is not best solution according to Marty’s comments. Lets use REST API then.
Documentation is here https://www.appveyor.com/docs/api/environments-deployments/#start-deployment
Check sample PowerShell function below. You can just call it with project-specific parameters when needed.
Note that you also have to set websitebuild.hostname to $(iis_site_name) in Deployment provider setting for sites use host header instead of * bindings on IIS, otherwise they will fight over the TCP port ;)
function start-appveyorDeploy
{
param (
[Parameter(Position=0, Mandatory=$true)]
[string]$projectSlug,
[Parameter(Position=1, Mandatory=$true)]
[string]$buildVersion,
[Parameter(Position=2, Mandatory=$true)]
[string]$siteName
)
$apiUrl = 'https://ci.appveyor.com/api'
# Replace this with your values
$token = <your_token>
$environmentName = <your_environmentName>
$accountName = <your_accountName>
$headers = #{
"Authorization" = "Bearer $token"
"Content-type" = "application/json"
}
$body = #{
environmentName=$environmentName
accountName=$accountName
projectSlug=$projectSlug
buildVersion=$buildVersion
environmentVariables =#{
iis_site_name=$siteName
}
}
$jsonBody = $body | ConvertTo-Json -Depth 3
Invoke-RestMethod -Uri "$apiUrl/deployments" -Headers $headers -Body $jsonBody -Method Post
}

Related

How can I delete an Azure DevOps build definition that it claims is retained by a release?

I'm trying to delete an Azure DevOps build definition but it won't let me since it says:
"One or more builds associated with the requested pipeline(s) are retained by a release. The pipeline(s) and builds will not be deleted."
However there's no obvious way to see what release is causing a build to be retained. I tried searching online, of course, but all the examples/screenshots of how to do this in the web UI are showing UI from several iterations ago of the Azure DevOps website so none of the controls look the same anymore. I don't see a lock icon anywhere, for example.
How can I find the releases that are holding onto these build definitions so I can delete them and then delete the build definition?
Thanks!
When you open the build pipeline to see its detailed build records, you can see the relevant release name and its link:
On old pipeline version, there had a lock icon which can obvious let us know it is retained. In fact, the lock icon not only means it is retained by release, manual build retain also show this icon. But, seems we missed this obvious icon while we expand the new sprint.
As a workaround to get list of builds which retained by release, here has a short script can help you achieve by using Rest api:
$token = "{PAT token}"
$url ="https://dev.azure.com/{org name}/{project name}/_apis/build/builds?api-version=5.1"
$token = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($token)"))
$response = Invoke-RestMethod -Uri $url -Headers #{Authorization = "Basic $token"} -Method Get
$results = $response.value | Where {$_.retainedByRelease -eq "true"} #|
Write-Host "results = $($results.id | ConvertTo-Json -Depth 100)"
First, use list builds api to get all builds that in current project. Then for the builds which retained by release, since there has a parameter can indicate it: retainedByRelease here I use $_.retainedByRelease -eq "true" to get the builds list which actual retained by release:
The above script is very universal, can be used in Powershell-ise and Powershell Command Line and the Powershell task of VSTS without change anything.
Update in 11/19:
Based on #Auth's comment, if want to get its associated release, the most easiest way is find the build, and then check its associate release pipeline as the screenshot shown I shared above.
If this does not satisfied what you want, and the previous API we used does not include any releases info in that, so here, you need use this API: Releases - Get Release:
GET https://vsrm.dev.azure.com/{org name}/{project name}/_apis/release/releases?sourceId={project id}:{build definition id}&api-version=5.1
In this API, you need specified the project id:build definition id to filter releases.
With the icon missing, will let the corresponding team know and try to add it in the future sprint.
I have had this issue when some one delete a repository that was associated with a "POC" pipeline. I was not able to delete the pipeline.
When I tried to delete the pipeline, I got following message:
One or more builds associated with the requested pipeline(s) are retained by a release. The pipeline(s) and builds will not be deleted.
I did following steps to delete the pipeline:
My Goal was to delete PublishBuildArtifacts-Infrastructure pipeline as shown in image. So I clicked on the pipeline.
I see list of Builds (Runs) that is associated with my pipeline. I click on the 3 dots and click on View Retention Lease.
Then I click Remove all.
Now you can delete your build.
Repeat steps 2-4 for each run. When you have delete all builds, now you can delete the pipeline by clicking on the 3 dot menu, as shown in the image below.
Enjoy!

Create bug in a different project when build fails

We have setup a build definition which creates a bug and assigns it to "requestor" incase the build fails. See below :
By default, the bugs are created in the same project. How do I change this to create the bugs in a different project ? The reason is, our work item tracking is in a different project than our Git repos (due to reshuffling and what not). Note that both the projects are under the same account.
Looking at the tool tip, it provides some sample fields, is there a corresponding field for "project" ? (System.Project ?!)
The Create work item on failure option in VSTS build can only create work item in the same project.
The workaround to create a Bug if build failed can be achieved by adding a PowerShell task. Detail steps as below:
1. Add a PowerShell in the end of your build definition
In the PowerShell script, you can create a Bug for a different project by REST API. Below is an example script:
$witType="Bug"
$witTitle="Build $(build.buildNumber) failed"
$u="https://account.visualstudio.com/DefaultCollection/project/_apis/wit/workitems/`$$($witType)?api-version=1.0"
$body="[
{
`"op`": `"add`",
`"path`": `"/fields/System.Title`",
`"value`": `"$($witTitle)`"
}
]"
$user = "user"
$token = "PAT"
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $user,$token)))
$result=Invoke-RestMethod -Method PATCH -Uri $u -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)} -ContentType "application/json-patch+json" -Body $body
2. Specify Run this task option as Even if a previous task has failed, unless the build was canceled
From Second task to the end task (PowerShell task), change Run this task option as Even if a previous task has failed, unless the build was canceled:
So when build fails, a Bug will be create to the different project as you specified in PowerShell script.
You can not do this out-of-the-box, but ...
You could do it using the API, something like this:
1 - Add a task to the build process to call a rest API and call the api to create a workitem. See https://www.visualstudio.com/en-us/docs/integrate/api/wit/work-items#create-a-work-item
2 - Change your agent phase to allow access to the token. Your task will need this.
3 - Change your task in Control Options and change the "Run this task" to "Custom condition" and fill the field with
eq(variables['Agent.JobStatus'], 'Failed')
Ref https://learn.microsoft.com/en-us/vsts/build-release/concepts/process/conditions?view=vsts
This should do it.

TFS API: Clone Build Definition Environment

I want to clone an existing environment within a build definition.
This is possible through the TFS GUI, but it doesn't seem like the API natively supports it.
So far, I've tried the following (in PowerShell):
Download build definition and deserialize
[void][System.Reflection.Assembly]::LoadFile("$pwd\Newtonsoft.Json.dll")
$TargetDefinitionName = "Name"
$TargetDefinitionID = ($DefinitionsOverview | Where-Object { $_.name -eq $TargetDefinitionName } | Select-Object -First 1).id
$TargetDefinitionURL = "$TfsUri/$TargetDefinitionID"
$TargetDefinitionJSON = Invoke-WebRequest -Uri "$TargetDefinitionURL" -UseDefaultCredentials
$DeserializedBuildDefinition = [Newtonsoft.Json.JsonConvert]::DeserializeObject($TargetDefinitionJSON.Content)
$DeserializedBuildDefinition.ToString()
Duplicate JSON block that represents environment, change unique properties
$NewEnvironmentString = $DeserializedBuildDefinition.environments[4].ToString()
$DeserializedBuildDefinition.environments.Add(5)
$DeserializedBuildDefinition.environments[5] = $NewEnvironmentString
$DeserializedBuildDefinition.environments[5].name.value = "NewEnvironment"
$DeserializedBuildDefinition.environments[5].rank.value = "6"
$DeserializedBuildDefinition.environments[5].id.value = "1665"
$DeserializedBuildDefinition.revision.value = "20"
Serialize JSON and post back (with new environment)
$SerializedBuildDefinition = [Newtonsoft.Json.JsonConvert]::SerializeObject($DeserializedBuildDefinition)
$PostData = [System.Text.Encoding]::UTF8.GetBytes($SerializedBuildDefinition)
$Headers = #{ "Accept" = "api-version=2.0-preview" }
$Response = Invoke-WebRequest -UseDefaultCredentials -Uri
$TargetDefinitionURL -Headers $Headers `
-Method Put -Body $PostData -ContentType "application/json"
$Response.StatusDescription
PROBLEM: $Response.StatusDescription gives "OK," but no new environment appears in the build definition.
One thought is that, in addition to 'name,' 'ID,' and 'rank, there are other values that need to be unique per environment that I'm missing.
I've also tried cloning the environment manually, saving the JSON representation of the definition, deleting the environment, and posting the JSON back. The new environment still doesn't show up.
Any help would be appreciated.
Future: Yaml based build definitions as code are well on their way, at least to VSTS. This won't help an immediate need but probably offers a better and more lightweight future approach.
When you copy an environment, you need to change "id", "name", and "rank". And the “id” need to be changed to "0". Also, try to use api-version=2.3-preview.1.
Then use the api of update a release definition to update the definition:
PUT http://tfs2015:8080/tfs/teamprojectcollection/teamProject/_apis/release/definitions?api-version=2.3-preview.1
{
……
}
I have tested with TFS 2015 Update 4. Api could clone an environment successfully.
I created a tool for cloning TFS builds, as (with TFS 2015 update 1 at least) the web 'clone' function does not clone the 'Repository' or 'Triggers' tab. Some of our build steps were a little complicated so I also wrote some simple transforms to automatically set the values according to the target branch.
I used the BuildHttpClient object in Microsoft.TeamFoundation.Build2.WebApi assembly to get the original build definition (BuildDefinition object, using GetDefinitionAsync() method), transform any build steps values, then call CreateDefinitionAsync() on BuildHttpClient to create the new build.
Works very reliably and under the hood it appears to use the TFS REST API.
If there was any interest, I'd be happy to create a more in depth tutorial on this.

Get Build status by Commit SHA1

Our team is using TeamCity for continuous integration and Git-TFS (TFS2015 RC) for versioning.
I want to check if a specific Commit has been built (successfully) by TeamCity and thus can be reintegrated.
It is possible to query TeamCity builds by using their REST API, e.g.:
GET http://teamcity:8000/guestAuth/app/rest/builds/18
with the result containing some xml that shows the git commit SHA:
<revisions count="1">
<revision version="96b1db05b64ecc895da070137e93cde3d2cadfa1">
<vcs-root-instance [...]/>
</revision>
</revisions>
The fact, that this information is in theory available, makes me hope that it is possible to query TeamCity builds by this particular information, like:
GET http://teamcity:8000/guestAuth/app/rest/builds/revision:96b1db05b64ecc895da[...]
but this yields a #400 BAD REQUEST response. I was not able to find out in the TeamCity 9 Documentation, whether this is possible or not. I would rather not iterate all builds to check if they contain this specific commit.
NOTE: This feature has now been implemented by JetBrains and is available in TeamCity 9.1 EAP4 now.
I don't believe this can be done without iteration, which is a little annoying
You can view changes by hash
/httpAuth/app/rest/changes?version:SHA_HASH
and you can find changes by build locator
/httpAuth/app/rest/changes?locator=build:(id:BUILD_ID)
but you can't go the other way, otherwise this could be done simply.
The buildLocator doesn't allow you to query using a revision dimension so I can't see any way around this
The following script may be of use to you if you haven't already written it yourself - save this to a file called get-build-status-by-git-commit.ps1 so it works with the sample at the end
# -----------------------------------------------
# Get Build Status By Git Commit
# -----------------------------------------------
#
# Ver Who When What
# 1.0 DevOpsGuys 01-07-15 Initial Version
# Script Input Parameters
param (
[ValidateNotNullOrEmpty()]
[string] $TeamCityServer = $(throw "-TeamCityServer is mandatory, please provide a value."),
[ValidateNotNullOrEmpty()]
[string] $ApiUsername = $(throw "-ApiUsername is mandatory, please provide a value."),
[ValidateNotNullOrEmpty()]
[string] $ApiPassword = $(throw "-ApiPassword is mandatory, please provide a value."),
[ValidateNotNullOrEmpty()]
[string] $GitSha = $(throw "-GitSha is mandatory, please provide a value.")
)
function Main()
{
$CurrentScriptVersion = "1.0"
$ApiCredentials = New-Object System.Management.Automation.PSCredential($ApiUsername, (ConvertTo-SecureString $ApiPassword -AsPlainText -Force))
Write-Host "================== Get Build Status By Git Commit - Version"$CurrentScriptVersion": START =================="
# Log input variables passed in
Log-Variables
Write-Host
# Set initial query url
$queryBuilds = "/httpAuth/app/rest/builds?fields=nextHref,build(id,status,revisions)"
while($queryBuilds)
{
$buildsToCheck = Api-Get "$TeamCityServer$queryBuilds"
$queryBuilds = $buildsToCheck.builds.nextHref;
foreach($build in $buildsToCheck.builds.build)
{
if ($build.revisions.revision.version -eq $GitSha) {
Write-Host "STATUS: "$build.status
Exit 0
}
}
}
Write-Host "================== Get Build Status By Git Commit - Version"$CurrentScriptVersion": END =================="
}
function Log-Variables
{
Write-Host "TeamCityServer: " $TeamCityServer
Write-Host "GitSha: " $GitSha
Write-Host "Computername:" (gc env:computername)
}
function Api-Get($Url)
{
Write-Host $Url
return Invoke-RestMethod -Credential $ApiCredentials -Uri $Url -Method Get -TimeoutSec 20;
}
Main
You can use this in the following way
.\get-build-status-by-git-commit.ps1 "http://teamcity:8000/" username password 96b1db05b64ecc895da070137e93cde3d2cadfa1
This is using httpAuth, but you could easily tailor the script to be using guest. I've used httpAuth in case it's of use to anyone else.
Hope this helps
I was looking for the same thing and stumbled upon this question. I don't know if you still need this but however I found the API in TeamCity v10. But you have to know your BuildTypeID for this
https://${teamcity.domain}/app/rest/buildTypes/id:${BuildTypeID}/builds/revision:${COMMIT_SHA}
You can find your build type id from TeamCity UI by going to your specific build and then
Edit configuration settings >> General Settings
And then the value in the Build configuration ID field.
Hope this helps.
The related feature request in TeamCity issue tracker: https://youtrack.jetbrains.com/issue/TW-40540. Vote for it.
The current workaround is to request builds with their revisions included into the response and then find the necessary builds on the client side.
The request can look like:
.../app/rest/builds?locator=buildType(id:XXX)&fields=build(id,href,revisions(revision))
Recent versions of teamcity allow the 'version' locator such that the following will return builds where the version SHA matches the query.
http://teamcity:8111/app/rest/changes?locator=version:SHA_HASH
I'm on TeamCity 2019.2, and I was able to directly search the builds endpoint for a specific commit:
TEAMCITY_URL/app/rest/builds/revision:SHA_HASH.
Not sure what version it was implemented in.

OctopusDeploy - Every website in the deploy has a different AppPool and Website name; how to deal; no other differences

I'm trying to setup a deploy process that targets 16 web sites each hosting an instance of the same application.
Websites and AppPools are named as such:
appServer1:
app10.site.com
app11.site.com
app12.site.com
app13.site.com
appServer2:
app20.site.com
app21.site.com
app22.site.com
app23.site.com
etc.
etc.
...with each website having a correspondingly named AppPool.
I am desperately trying to determine how to use a single Deploy NuGet Package step to target all of these websites/app pools using variables and a combination of powershell scripts if possible.
I'd like to have a single step where I can variable substitute the website and app pool names. As this is the only difference. I basically need the equivalent of being able to loop the nuget package step passing it a list of website and app pool names. I cannot simply use variables because I can only resolve to the machine level with variable scoping.
Create list of all Website and AppPool names, iterate them passing each value to a Step for execution. ForEach processing step for lack of better words.
I do have the ability to rename the AppPools if need be for a more consistent pattern, but I cannot change the website names
Any ideas would be greatly appreciated.
http://help.octopusdeploy.com/discussions/questions/3481-every-website-in-the-deploy-has-a-different-apppool-and-website-name-how-to-deal-no-other-differences
There's a lot to your question, but I'm going to take a stab at explaining our approach, in hopes of jogging your creative juices.
tl;dr
simply put, use your own powershell scripts to install the web-application. In there you can set the app pool name on a per website basis
For starters, we do do a separate deployment step for each project. The scripts we use will allow you to do all deployments from a single deploy.ps1 (including unique appPool names), but we find that it really helps keep each deployment nice and lean, and easy to manage. Each project get's it's own nupkg and therein contains the predeploy.ps1, deploy.ps1, and postdeploy.ps1 as well as a folder of build/deploy scripts that we've open sourcesd, and a folder of environment config xml files.
A sample of an environment config would be this. The name is simply [envName].xml
<!-- environments\Production.xml -->
<environmentSettings>
<webSites>
<app>
<physicalPathRoot>c:\inetpub</physicalPathRoot>
<physicalFolderPrefix>appname</physicalFolderPrefix>
<siteProtcol>https</siteProtcol>
<siteName>appname.tld</siteName>
<siteHost>appname.tld</siteHost>
<portNumber>443</portNumber>
<appPath>/</appPath>
<appPool>
<name>appname.tld</name>
<!-- valid identityTypes are: [LocalSystem, LocalService, NetworkService, SpecificUser, ApplicationPoolIdentity] -->
<identityType>NetworkService</identityType>
<!-- Set this value to the User the Service will run under in the format DOMAIN\username -->
<!-- If Running as 'NetworkService' then 'NT AUTHORITY\Network Service' is used -->
<userName>NT AUTHORITY\Network Service</userName>
<!-- Leave blank unless using SpecificUser -->
<password></password>
<maxWorkerProcesses>5</maxWorkerProcesses>
</appPool>
</app>
</webSites>
<serverDatabase>
<name>database_name</name>
<connectionString>REPLACED BY OCTOPUS</connectionString>
<providerName>System.Data.SqlClient</providerName>
</serverDatabase>
</environmentSettings>
You can see in the corresponding Get-EnvironmentSettings.ps1 where we load up the config, and then update it with any Octopus variables. This is the trickiest part, because we use dot-Notation to update the paths (case sensitive).
Our octopus variables really only contain information that is secret, as everything else lives in [environment].xml
| Name | Value | Scope
--------------------------------------------------------------------------
| webSites.app.appPool.password | supersecret | Production
So now a typical deployment script simply imports the modules, grab environmentSettings, update config, and install the web app.
# Top of the script, get Octopus environment and version
param(
[string] $version = $OctopusPackageVersion,
[string] $environment = $OctopusEnvironmentName
)
# Make sure a failed deployment actually fails
$ErrorActionPreference = "Stop"
# Import the modules
$currentDir = Split-Path $script:MyInvocation.MyCommand.Path
$moduleDir = "$currentDir\modules"
Import-Module BuildDeployModules
# Grab the environment settings
$environmentSettings = Get-EnvironmentSettings $environment "//environmentSettings"
$databaseSettings = $environmentSettings.serverDatabase
$websiteSettings = $environmentSettings.webSites.app
# update the config
Update-XmlConfigValues $currentDir\website\Web.config "//appSettings/add[#key='databaseName']" $($databaseSettings.name) "value"
Update-XmlConfigValues $currentDir\website\Web.config "//connectionStrings/add[#name='databaseConnection']" $($databaseSettings.connectionString) "connectionString"
Update-XmlConfigValues $currentDir\website\Web.config "//connectionStrings/add[#name='databaseConnection']" $($databaseSettings.providerName) "providerName"
# Install the web application
Install-WebApplication $environment $websiteSettings $version "anonymousAuthentication"
In doing all of this, the web application is installed into IIS with a specific application pool, and appropriate config transforms without relying on any unknowns.
Our nupkg structure looks something like this
appname.1.2.3.4.nupkg
environments
dev.xml
staging.xml
qual.xml
production.xml
modules
[all of our build modules]
website
[all of our website files]
This is super repeatable, easy to maintain, and easy to edit config. Hope it helps