As they are, I need to change the directory where my repository is cloned, in the documentation I saw that there is a variable Pipeline.Workspace but I can not change it, I'm working with a self-hosted agent
Change Variable Pipeline.Workspace in Azure Devops
To change the the default work folder _work that Azure Devops agents use when building a pipeline, you can open the hidden.agent file in the installation directory of the private agent and change the workFolder to the place you want:
{
"agentId": 9,
"agentName": "VsAgent1",
"poolId": 10,
"serverUrl": "https://dev.azure.com/MyXXXXOrganization/",
"workFolder": "_work"
}
As test, I use bash task to output the value of variable Pipeline.Workspace with the command line echo $(Pipeline.Workspace):
The default value is C:\VS2017Agent\_work\14:
Then I change the workFolder from _work to D:\\tfsagent\\_work in the .agent file and run the build pipeline again:
The changed value is D:\tfsagent\_work\1:
Check the document How to change the TFS Agent _work folder for some more details.
Hope this helps.
Related
I am trying to create a 'Synapse Managed private endpoint' to an Azure storage account via a 'Azure cli' task as a step in a pipeline. I want to create the MPE automatically.
The pipeline step calls a power-shell script with parameters. The script is located in source control. Calling the script and passing in parameter values is working fine.
Within the powershell script the following happens...
Get the json template (see below) from source control - this step works.
In the powershell script subsitute the json fields enclosed in <...> with a the parameter values passed in to the power-shell script - this works fine. The converted json is shown in the below screenshot.NB: sensitive values have been readacted here but look correct...
The line in the power-shell that's raising the error is...
az synapse managed-private-endpoints create --workspace-name "$pSynapseWorkspaceName" --pe-name "$pPrivateEndpointName" --file $mpeArmJson --debug --verbose
I think it's to do with the json-string parmater $mpeArmJson and double-quotes - this is what I need help solving ???
The value of $mpeArmJson which the value at this point is (note sensitive values have been readacted here but look correct)...
{
"name": "dds2-datalake-endpoint",
"properties": {
"privateLinkResourceId": "/subscriptions/<redacted subscription id>/resourceGroups/dds2-data-tst-rg/providers/Microsoft.Storage/storageAccounts/dds2datatstdlksa",
"groupId": "dfs",
"fqdns": [
"<redacted-storage-account>.dfs.core.windows.net"
]
}
}
In my Azure devOps pipeline I have created a 'Azure Powershell' task. The task calls a PowerShell script stored in source-control - taking in parameters.
Inside the powershell script I am calling...
New-AzSynapseManagedPrivateEndpoint -WorkspaceName "$pSynapseWorkspaceName" -Name "$pPrivateEndpointName" -DefinitionFile "$tmpDir"
NB: the devOps pipeline runs under a Az 'service principal' which creates the Synapse workspace and in doing so gets the 'owner' and also a 'synapse admin.' permissions automatically set in IAM.
The 'service principal' also needs 'blob storage data contributor' on the main storage account linked to Synapse.
I'm a newbie to both Azure DevOps and Terraform but, I'm trying to deploy a pipeline using a YAML file.
I have tried to run a terraform plan using a YAML file and passing variables (from AZ DevOps) but, I got the following error:
2021-11-24T18:39:46.4604561Z Error: "name" may only contain alphanumeric characters, dash, underscores, parentheses and periods
2021-11-24T18:39:46.4604832Z
2021-11-24T18:39:46.4605940Z on modules/aks/main.tf line 2, in resource "azurerm_resource_group" "aks-resource-group":
2021-11-24T18:39:46.4606436Z 2: name = var.resource_group_name
2021-11-24T18:39:46.4606609Z
2021-11-24T18:39:46.4606722Z
2021-11-24T18:39:46.4606818Z
2021-11-24T18:39:46.4607525Z Error: Error: Subnet: (Name "#{vnet_subnet_name}#" / Virtual Network Name "#{vnet_name}#" / Resource Group "RG-XX-XXXX-XXXXX-001") was not found
2021-11-24T18:39:46.4608006Z
2021-11-24T18:39:46.4608580Z on modules/aks/main.tf line 16, in data "azurerm_subnet" "subnet-project":
2021-11-24T18:39:46.4609335Z 16: data "azurerm_subnet" "subnet-project" {
The 'name' has the following format at the Variable group in the Azure DevOps UI:
RG-XX-XXXX-XXXXX-001
This is the snippet of where I included the replace token at the YAML file:
displayName: 'Replace Secrets'
inputs:
targetFiles: |
variables.tfvars
encoding: 'utf-8'
actionOnMissing: fail
tokenPattern: #{MyVar}#
And this is a sample of the variables I have in a variable group:
variable-group-sample
Also, I replace the terraform.tfvars file with something like this:
resource_group_name = "#{resource_group_name}#"
I have checked the name inserted at the UI several times but I feel the error is pointing to something else I cannot see.
Have anyone experienced something related to this error?
Thank you in advance!
tokenPattern: #{MyVar}#
It is looking for the pattern #{MyVar}# to replace. Not "something contained between #{ and }#, but the actual value #{MyVar}#. I'm guessing it's expecting a regular expression, but I'm not familiar with that task.
So the end result is that your #{token values}# aren't getting replaced.
Assuming you're using https://marketplace.visualstudio.com/items?itemName=qetza.replacetokens, you probably want to specify tokenPrefix: #{ and tokenSuffix: }# instead of using tokenPattern.
Now, having said that...
There is no reason for you to be using token replacement on a tfvars file. You should create different tfvars files for each environment, then pass in a tfvars file via the -var-file argument to Terraform. Secrets can be passed in on the command line via -var 'foo=bar'
Storing variables that represent application or deployment configuration in Azure DevOps (or GitHub, or any other CI system) is a big, big anti-pattern, because it's tightly coupling your deployment process to a particular platform. If you're sourcing all of your variables from Azure DevOps, you can't easily test locally or migrate to a different CI/CD provider like GitHub Actions in the future.
For values that shouldn't be in source control, such a secrets, you should use a secret provider like Azure KeyVault and integrate it with your application (or, in this case, use a data resource in Terraform to pull the necessary secrets automatically at deployment time).
I am creating the CI/CD pipeline in Azure DevOps for .NetCore API, and I have multiple deployment stages like QA, UAT and Production. Every stages has its own setting like database connectionstrings and many other, and I have one appsettings.json file in which I declared these settings, so its very hard work to change appsettings.json file each time in deployment, so my question is that how I can use environment variables in CI/CD pipelines to avoid all this stuff,
As I Just need to define the variables values in variables section and CD pipeline can automatically pick it from my appsetting.json file.
appsetting.json File
"Logging": {
"LogLevel": {
"Default": "Information",
"Microsoft": "Warning",
"Microsoft.Hosting.Lifetime": "Information"
}
},
"AllowedHosts": "*",
"SubscriptionClientName": "api",
"MessageBrokerRetryCount": 5,
"ConnectionStrings": {
"CaseDBConnectionString": "Server=db-server;Database=api_db;User ID=user;pwd =Password;"
},
//For Dev
"MessageBrokerConnection": "ip",
"MessageBrokerUserName": "john",
"MessageBrokerPassword": "john"
}
We typically use the "Replace Tokens" task in our releases, and then set the variables in our release pipeline for each environment that will replace the tokens in our appsettings.json file. Very simple to use and set up.
UPDATE:
In your appsettings.json file, you would do something like:
"Username": "#{AccountUser}#"
Then in your pipeline, you would go into your release and create a variable called "AccountUser". Then you use the Replace Tokens task in your release as the first step, and it will replace #{AccountUser}# in your appsettings.json file with whatever the value of the variable is. Set different variables for each Scope (dev, stage, production or whatever you call your environments in your release) so that your appsettings.json file gets the appropriate value per environment.
a more reasonable approach would be to use application settings of you App Service, they would override values in appsettings.json\web.config
https://learn.microsoft.com/en-us/azure/app-service/configure-common
this might require slight code changes on your part:
https://learn.microsoft.com/en-us/azure/app-service/containers/configure-language-dotnetcore#access-environment-variables
I created a new build with Azure Pipelines (Azure DevOps) and it worked really well.
Usually, you use $(Rev:.r) to get the revision in the build. Unfortunately, it seems the variable isn't replaced/set in the build steps. The only place where you can use it is the name: property in the YAML document.
Now I set it in the name and extract it in some PowerShell, which isn't necessary if you can get it via an environment variable.
How do I get the Revision (like $(Rev)) in the new builds (outside of the name: property in the YAML document)?
(The Build Agents running on-premise, inside Docker - but this shouldn't affect the things above)
You can't get the revision number without parsing, it is not stored as a separate field somewhere or in an environment variable.
The $(Rev:.r) portion instructs Azure DevOps to come up with the first number that makes the build number unique (and, in that specific example, put a dot in front of it).
Like you said, the only way is to use PowerShell script to get the value:
$buildNumber = $Env:BUILD_BUILDNUMBER
$revision= $buildNumber.Substring($buildNumber.LastIndexOf('.') + 1)
Edit:
You can install the Get Revision Number extension that does it.
Another possible solution to the above problem could be to use counter expression for ex: we difine the variable and use it in a task to build nuget package.
variables:
counterVar: $[counter($(versionVariable),0)]
.......
- task: CmdLine#2
inputs:
script: >
nuget pack ClassLibrary1/ClassLibrary1.csproj
-OutputDirectory $(Build.ArtifactStagingDirectory)
-NonInteractive
-Properties Configuration=release
-Version $(versionVariable).$(counterVar)
-Verbosity Detailed
-IncludeReferencedProjects
Here versionVariable is a custome variable defined in pipelines->variables.And the seed value is 0(2nd param to counter).
It works as below
Let's assume the versionVariable is 1.19
Build Run 1 counterVar will be 0.
Build Run 2 counterVar will be 1.
Now say we change the versionVariable to 1.20
Build Run 3 counterVar will be 0.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops
Check the counter expression in above link it reset its value for diff prefix.
P.S. Benefit of using counter over $(Rev:r) is that it can start from 0 unlike $(Rev:r)
In previous versions of TFS (before 2015), there was a build environment variable for PowerShell called: TF_BUILD_DROPLOCATION, which gave the The location of the drop:
https://msdn.microsoft.com/library/hh850448%28v=vs.120%29.aspx.
I can't find the equivalent variable in TFS 2017.
What is the best practice to get it?
With Build agent tasks taking over things are different. What I do to see the various build environment variables is to make a simple batch file containing this:
SET C:\temp\EnvVars.txt
That'll produce a quick list of what is available.
Here's what I see with the TFS 2017 build agent:
agent.jobstatus=Succeeded
AGENT_BUILDDIRECTORY=C:\Agent\_work\2
AGENT_HOMEDIRECTORY=C:\Agent
AGENT_ID=2 AGENT_JOBNAME=Build
AGENT_JOBSTATUS=Succeeded
AGENT_MACHINENAME=BUILDMACHINE
AGENT_NAME=BUILDMACHINE
AGENT_OS=Windows_NT
AGENT_ROOTDIRECTORY=C:\Agent\_work
AGENT_SERVEROMDIRECTORY=C:\Agent\externals\vstsom
AGENT_TEMPDIRECTORY=C:\Agent\_work\_temp
AGENT_TOOLSDIRECTORY=C:\Agent\_work\_tool
AGENT_VERSION=2.122.1
AGENT_WORKFOLDER=C:\Agent\_work
BUILD_ARTIFACTSTAGINGDIRECTORY=C:\Agent\_work\2\a
BUILD_BINARIESDIRECTORY=C:\Agent\_work\2\b
BUILD_BUILDID=2036
BUILD_BUILDNUMBER=Database Build_20190708.2
BUILD_BUILDURI=vstfs:///Build/Build/2036
BUILD_CONTAINERID=2281
BUILD_DEFINITIONNAME=Database Build
BUILD_DEFINITIONVERSION=17
BUILD_QUEUEDBY=Smith, John
BUILD_QUEUEDBYID=8c588342-b87a-40cb-9b8c-a0ed10b57a3f
BUILD_REASON=Manual
BUILD_REPOSITORY_CLEAN=false
BUILD_REPOSITORY_GIT_SUBMODULECHECKOUT=False
BUILD_REPOSITORY_ID=$/
BUILD_REPOSITORY_LOCALPATH=C:\Agent\_work\2\s
BUILD_REPOSITORY_NAME=Collection
BUILD_REPOSITORY_PROVIDER=TfsVersionControl
BUILD_REPOSITORY_TFVC_WORKSPACE=ws_2_2
BUILD_REPOSITORY_URI=http://TFSSERVER:8080/tfs/Project/
BUILD_REQUESTEDFOR=Smith, John
BUILD_REQUESTEDFOREMAIL=John.Smith#Mailinator.com
BUILD_REQUESTEDFORID=7a588222-b66a-40ee-9b2a-a0ba10b12a3f
BUILD_SOURCEBRANCH=$/Collection/Project/Code
BUILD_SOURCEBRANCHNAME=Code
BUILD_SOURCESDIRECTORY=C:\Agent\_work\2\s
BUILD_SOURCEVERSION=9811
BUILD_SOURCEVERSIONAUTHOR=Smith, John
BUILD_SOURCEVERSIONMESSAGE=Added missing permission
BUILD_STAGINGDIRECTORY=C:\Agent\_work\2\a
You can list all Environment Variables with the following command:
get-childitem ENV:\
I am assuming you could create a simple build job that executes this and then look at the console output to determine what the name is of the Environment Variable you need.