Use variable from AWSCLI in azure pipelines for script - azure-devops

I have a build process where I need to use a token, received through the AWSCLI. So far I have connected aws to my azure pipelines but I am having trouble setting up my yaml.
I want to fetch the relevant token to use it later as a variable in my script.
As you can see in my yaml I am running a powershell script with codeartifact and I am saving the value to my myOutputVar. The powershell script does not throw an error.
However, later when I run the building script that variable is not present resulting in ECHO is off.
How can I ensure the value received in the task can be used later in the script/build part?
trigger:
- azure-pipelines
pool:
vmImage: windows-latest
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- task: AWSPowerShellModuleScript#1
inputs:
awsCredentials: 'AWS Connection'
regionName: 'eu-central-1'
scriptType: 'inline'
inlineScript: '##vso[task.setvariable variable=myOutputVar;]aws codeartifact get-authorization-token --domain somedomain --domain-owner 444.... --query authorizationToken --output text; '
- script: |
echo %myOutputVar%
npm ci
npm run build
displayName: 'npm install and build'

Your inline script can be multiple lines, and since this is PowerShell you can do something like:
inlineScript: |
$authToken = aws codeartifact get-authorization-token `
--domain somedomain `
--domain-owner 444.... `
--query authorizationToken `
--output text
Write-Host "##vso[task.setvariable variable=myOutputVar;]$authToken"

Related

AzureStaticWebApp#0 not recognizing deployment token from variable

Hi have the following code that deploys an artifact to an Azure Static Web App:
...
variables:
- name: staticWebAppDeploymentToken
...
# This steps reads the deployment token of the static web app and assigns it on a variable
- task: AzureCLI#2
displayName: 'Retrieve static web app deployment token'
inputs:
azureSubscription: xxxx
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
output=$(az staticwebapp secrets list --name xxxx-xxxx-$(environment) | jq .properties.apiKey)
echo "##vso[task.setvariable variable=staticWebAppDeploymentToken;]$output"
- task: AzureStaticWebApp#0
inputs:
output_location: '/'
cwd: '$(Pipeline.Workspace)/artifact'
skip_app_build: true
azure_static_web_apps_api_token: $(staticWebAppDeploymentToken)
I get the error:
I've set the System.Debug variable to true, and I see the value is set in the variable. I've also printed the variable and the value is there.
I can't understand what I'm doing wrong. What is the correct way to set a variable in bash and use it on another non-bash step?
I've tried hardcoding the value and also passing it as a parameter from the library, and that works, but that is not what I want.
Test the same script to get the token and pass it to Azure Static web APP task, I can reproduce the same issue.
The root cause of this issue is that when you set the variable in Azure CLI task with the command, the pipeline variable will contain "". For example: "tokenvalue".
The expected deployment token in Azure Static web APP task, it will not contain double quotes.
To solve this issue, you need to add a step in Azure CLI task to remove the double quotes.
Here is an example:
steps:
- task: AzureCLI#2
displayName: 'Azure CLI '
inputs:
azureSubscription: xx
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
output=$(az staticwebapp secrets list --name kevin0824 | jq .properties.apiKey)
var2=`sed -e 's/^"//' -e 's/"$//' <<<"$output"`
echo $var2
echo "##vso[task.setvariable variable=staticWebAppDeploymentToken; isOutput=true]$var2"
- task: AzureStaticWebApp#0
displayName: 'Static Web App: '
inputs:
app_location: /
api_location: api
skip_app_build: false
skip_api_build: false
is_static_export: false
verbose: false
azure_static_web_apps_api_token: '$(staticWebAppDeploymentToken)'

Azure devops deploy to function app with access restrictions

I am trying to deploy a function app via an Azure DevOps pipeline, however I am receiving the following error:
##[error]Failed to deploy web package to App Service.
##[error]To debug further please check Kudu stack trace URL : $URL_REMOVED
##[error]Error: Error: Failed to deploy web package to App Service. Ip Forbidden (CODE: 403)
From some googling a suggested solution seems to be to whitelist agent IP before the deployment, and then remove it after. I have added this to my pipeline, and I can see the agent IP get added to access restrictions, however the deployment still fails.
Here is my pipeline file:
# Node.js Function App to Linux on Azure
# Build a Node.js function app and deploy it to Azure as a Linux function app.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/javascript
trigger:
- main
variables:
# Azure Resource Manager connection created during pipeline creation
azureSubscription: 'xxx'
# Function app name
functionAppName: 'xxx'
# Environment name
environmentName: 'xxx'
# Agent VM image name
vmImageName: 'ubuntu-latest'
stages:
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: NodeTool#0
inputs:
versionSpec: '10.x'
displayName: 'Install Node.js'
- script: |
if [ -f extensions.csproj ]
then
dotnet build extensions.csproj --runtime ubuntu.16.04-x64 --output ./bin
fi
displayName: 'Build extensions'
- script: |
npm install
npm run build --if-present
npm run test --if-present
displayName: 'Prepare binaries'
- task: ArchiveFiles#2
displayName: 'Archive files'
inputs:
rootFolderOrFile: '$(System.DefaultWorkingDirectory)'
includeRootFolder: false
archiveType: zip
archiveFile: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
replaceExistingArchive: true
- upload: $(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip
artifact: drop
- stage: Deploy
displayName: Deploy stage
dependsOn: Build
condition: succeeded()
jobs:
- deployment: Deploy
displayName: Deploy
environment: $(environmentName)
pool:
vmImage: $(vmImageName)
strategy:
runOnce:
deploy:
steps:
- task: AzureCLI#2
inputs:
azureSubscription: '$(azureSubscription)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
agentIP=$(curl -s https://api.ipify.org/)
az functionapp config access-restriction add -g xxx -n xxx --action Allow --ip-address $agentIP --priority 200
- task: AzureFunctionApp#1
displayName: 'Azure Functions App Deploy: xxx'
inputs:
azureSubscription: '$(azureSubscription)'
appType: functionAppLinux
appName: $(functionAppName)
package: '$(Pipeline.Workspace)/drop/$(Build.BuildId).zip'
Is anyone able to advise where I am going wrong?
I've had a simmilar issue while adding the agent IP to the network restrictions of an storage account (using Powershell but you'll understand the idea), we added a 60s sleep to be sure that the setting are taken into account by Azure.
$sa_name = "sapricer$env_prefix"
if ($null -ne (Get-AzStorageAccount -ResourceGroupName $sa_rg -AccountName $sa_name -ErrorAction Ignore)) {
Write-Output "Storage account '$sa_name' exists"
if ($enable) {
Write-Output "Add ip rule for $current_ip on $sa_name..."
Add-AzStorageAccountNetworkRule -ResourceGroupName $sa_rg -AccountName $sa_name -IPAddressOrRange $current_ip
}
else {
Write-Output "Remove ip rule for $current_ip on $sa_name..."
Remove-AzStorageAccountNetworkRule -ResourceGroupName $sa_rg -AccountName $sa_name -IPAddressOrRange $current_ip
}
}
Start-Sleep -Seconds 60
I found the solution to this.
Function Apps have two IP Restriction sections, one for the App and one for the SCM site. The SCM site is the one that requires the IP to be whitelisted in order for the deployment to work:
az functionapp config access-restriction add --scm-site true -g xxx -n xxx --action Allow --ip-address $agentIP --priority 200
You can deploy Azure function app to azure devops pipeline using azure function app task from Azure devops pipeline tasks
Here is the sample snippet for deploying azure function app
variables:
azureSubscription: Contoso
# To ignore SSL error, uncomment the below variable
# VSTS_ARM_REST_IGNORE_SSL_ERRORS: true
steps:
- task: AzureFunctionApp#1
displayName: Azure Function App Deploy
inputs:
azureSubscription: $(azureSubscription)
appName: samplefunctionapp
package: $(System.DefaultWorkingDirectory)/**/*.zip
Here is the Microsoft Document for deploying azure function app.

Pass Variable Group as Dictionary To Python Script

I have a variable group that i'm using from a python script. Something like this:
- task: PythonScript#0
inputs:
scriptSource: 'inline'
script: |
print('variableInVariableGroup: $(variableInVariableGroup)')
I'd like to write my script so that I can iterate over the variable group without explicitly referencing individual variables. Is there a way to feed in the entire variable group to the script as a dictionary or something similar?
You could not do that directly, for the workaround is to get the vars via azure cli and set with task variable, then get them in the python script task.
Something like below:
# 'Allow scripts to access the OAuth token' was selected in pipeline. Add the following YAML to any steps requiring access:
# env:
# MY_ACCESS_TOKEN: $(System.AccessToken)
# Variable Group 'vargroup1' was defined in the Variables tab
resources:
repositories:
- repository: self
type: git
ref: refs/heads/testb2
jobs:
- job: Job_1
displayName: Agent job 1
pool:
vmImage: ubuntu-20.04
steps:
- checkout: self
persistCredentials: True
- task: PowerShell#2
name: TestRef
displayName: PowerShell Script
inputs:
targetType: inline
script: >-
echo $(System.AccessToken) | az devops login
$a=az pipelines variable-group variable list --org 'https://dev.azure.com/orgname/' --project testpro1 --group-id 3 --only-show-errors --output json
echo "$a"
echo "##vso[task.setvariable variable=allvars;isOutput=true]$a"
- task: PythonScript#0
displayName: Run a Python script
inputs:
scriptSource: inline
script: "b=$(TestRef.allvars)\nprint(b)\n\n "
...

Unable to run az pipeline commands within Azure DevOps Task

Trying to dynamically retrieve all the variables from a variable group via Azure DevOps task in a YAML Pipeline. Originally tried leveraging the AzureCLI#2 task with the following code to retrieve the variableGroupID which would be used to get the variables inside of it:
$variableGroupId = $(az pipelines variable-group list --org $(System.CollectionUri) --project $(System.TeamProject) --query "[?name=='{{ parameters.variableGroupName }}'].id" -o tsv)
This command works locally but not when executing on a MS hosted agent like this:
parameters:
variableGroupName: ''
steps:
- task: AzureCLI#2
displayName: Azure CLI
inputs:
azureSubscription: ${{ parameters.azureSubscriptionName }}
scriptType: pscore
scriptLocation: inlineScript
inlineScript: |
az upgrade
$variableGroupId = $(az pipelines variable-group list --org $(System.CollectionUri) --project $(System.TeamProject) --query "[?name=='{{ parameters.variableGroupName }}'].id" -o tsv)
write-Host $variableGroupId
$variables = $(az pipelines variable-group variable list --group-id $variableGroupId --org $(System.CollectionUri) --project $(System.TeamProject) -o yaml)
write-Host $variables
This fails with the error:
Before you can run Azure DevOps commands, you need to run the login command (az login if using AAD/MSA identity else az devops login if using PAT token) to setup credentials. Please see https://aka.ms/azure-devops-cli-auth for more information
I have opened up an issue
In the meantime, I tried to run the commands to install the necessary pieces via scripts
strategy:
runOnce:
deploy:
steps:
- task: AzureRmWebAppDeployment#3
inputs:
azureSubscription: Example - Dev
WebAppName: wapp-Example-dev-eus
Package: $(Pipeline.Workspace)/drop/Web.Example.zip
TakeAppOfflineFlag: True
- task: UsePythonVersion#0
inputs:
versionSpec: '3.x'
architecture: 'x64'
- task: CmdLine#2
displayName: 'Upgrade pip'
inputs:
script: python -m pip install --upgrade pip
- task: CmdLine#2
displayName: 'upgrade azure cli'
inputs:
script: pip install --pre azure-cli --extra-index-url https://azurecliprod.blob.core.windows.net/edge
- task: CmdLine#2
displayName: 'Show Azure CLI version'
inputs:
script: az --version
- task: CmdLine#2
displayName: 'Install Azure DevOps Extension'
inputs:
script: az extension add -n azure-devops
- task: CmdLine#2
env:
AZURE_DEVOPS_CLI_PAT: $(patCredential)
displayName: 'Login Azure DevOps Extension'
inputs:
script: echo ${AZURE_DEVOPS_CLI_PAT} | az devops login
- task: CmdLine#2
displayName: 'Show List of Variables'
inputs:
script: |
$variableGroupId = $(az pipelines variable-group list --org $(System.CollectionUri) --project $(System.TeamProject) --query "[?name=='{{ parameters.variableGroupName }}'].id" -o tsv)
write-Host $variableGroupId
$variables = $(az pipelines variable-group variable list --group-id $variableGroupId --org $(System.CollectionUri) --project $(System.TeamProject) -o yaml)
write-Host $variables
However, when using both latest Ubuntu agents and those designated in the doc get an error:
WARNING: Failed to store PAT using keyring; falling back to file storage.
WARNING: You can clear the stored credential by running az devops logout.
WARNING: Refer https://aka.ms/azure-devops-cli-auth to know more on sign in with PAT.
I have opened up an issue with the documentation team as at the very least the provided steps do not work. Any assistance would be appreciated!
I was getting the same error, and was able to get mine working by adding:
echo $(System.AccessToken) | az devops login
to the top of my inline script. Here's what it looks like:
variables:
variableGroupName: 'my-variable-group'
...
- task: AzureCLI#2
displayName: 'Set environment variables'
inputs:
azureSubscription: '$(azureSubscription)'
scriptType: 'pscore'
scriptLocation: 'inlineScript'
inlineScript: |
echo $(System.AccessToken) | az devops login
$groupId = (az pipelines variable-group list `
--organization $(System.CollectionUri) `
--project $(System.TeamProject) `
--group-name $(variableGroupName) | ConvertFrom-Json).id
...
You can use the REST API instead of Azure CLI to get the information. It can be used with the standard tools already present on the Microsoft Hosted agents. It requires only vanilla powershell or powershell core, meaning to works on both windows and linux agents. The below example was successfully tested on windows-latest/windows-2019 and ubuntu-latest/ubuntu-20.04
The approach is the same as with Azure CLI.
List all available groups filtered by name to retrieve the variable group in question
Get all variables in the variable group using the variable group id from step
In fact, the pipeline also has an out of the box PAT token available with read access to variable groups. It is stored in the variable System.AccessToken. Using that instead of a manually managed one will further simplify things.
The script below is executed in a pwsh step, which is the built in Powershell task in Powershell core mode
- pwsh: |
# Construct PAT authentication header
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "user",$env:SYSTEM_ACCESSTOKEN)))
$headers = #{Authorization=("Basic {0}" -f $base64AuthInfo)}
# Retrieve variable group id. Filter the result by setting the groupName query parameter
$variableGroupId = $(Invoke-RestMethod -Headers $headers "$(System.CollectionUri)$(System.TeamProject)/_apis/distributedtask/variablegroups?groupName=${{ parameters.variableGroupName }}&api-version=6.0-preview.2").value[0].id
# Retrieve variables in variable group with id $variableGroupId
$variables = $(Invoke-RestMethod -Headers $headers "$(System.CollectionUri)$(System.TeamProject)/_apis/distributedtask/variablegroups/${variableGroupId}?api-version=6.0-preview.2").variables
#Print variables as json (for demo purpose)
$variables | ConvertTo-Json
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
displayName: 'Retrieve variables'
Testing the above pipeline in a project on a variable group with two variables yields the following output:
{
"Variable 1": {
"value": "Value 1"
},
"Variable 2": {
"value": "Value 2"
}
}

Azure DevOps pipelines passing environment variables into PS Script on an Ubuntu Host

I have a Stage that runs my Terraform Code, but I need to whitelist the agent IP on my Azure SQL server, so I have a PowerShell script that can handle this, however, the environment variables inside the script (for instance $env:company_name) are all showing as null I believe because its bash. Not sure....the part of YAML is shown below. Any ideas on how I can accomplish this? Thanks
******************** YAML from pipeline**************************
steps:
- checkout: self
- task: Bash#3
displayName: 'Install AZ Modules'
inputs:
targetType: 'inline'
script: |
sudo /usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -File "$(Build.Repository.LocalPath)/STAGE2/Scripts/buildazmodule.ps1"
- task: PowerShell#2
displayName: 'Add Firewall Rules'
inputs:
targetType: filePath
filePath: './STAGE2/Scripts/firewallrules.ps1'
- task: ms-devlabs.custom-terraform-tasks.custom-terraform-installer-task.TerraformInstaller#0
inputs:
terraformVersion: '0.12.28'
- script: terraform version
displayName: 'Terraform Version'
- script: az cloud set --name $(cloud)
displayName: 'Set Cloud'
- script: az login --service-principal -u $(client_id) -p $(client_secret) --tenant $(tenant_id)
displayName: 'Log Into Azure'
- script: terraform init -backend-config=resource_group_name=$(sg_resource_group) -backend-config="storage_account_name=$(sg_name)" -backend-config="container_name=$(blob_storage)" -backend-config="access_key=$(sg_accesskey)" -backend-config="key=$(state_file)" -backend-config="environment=$(cloud_environment_name)"
displayName: 'Terraform Init'
workingDirectory: $(System.DefaultWorkingDirectory)/STAGE2
- script: terraform plan -var="client_id=$(client_id)" -var="client_secret=$(client_secret)" -var="tenant_id=$(tenant_id)" -var="subscription_id=$(subscription_id)" -var="environment=$(cloud_environment_name)" -var="company_name=$(company_name)" -var="cloudsitename=$(cloudsitename)" -var="envtype=$(envtype)" -var="builddate=$(builddate)" -var="is_public=$(is_public)" -var="region=$(region)" -var="os_image_skey=$(os_image_skey)" -var="is_osmanaged_ad=$(is_osmanaged_ad)" -var="remote_tenant=$(remote_tenant)" -var="is_fedramp=$(is_fedramp)" -var="level=$(level)" -var="onestream_version=$(onestream_version)" -out="out.plan"
displayName: 'Terraform Plan'
workingDirectory: $(System.DefaultWorkingDirectory)/STAGE2
- script: terraform apply out.plan
displayName: 'Terraform Apply'
workingDirectory: $(System.DefaultWorkingDirectory)/STAGE2
************************ Part of script code***********************
$envtype = $env:envtype
$CompanyName = $env:company_name
$CompanyNameTemp = $CompanyName.ToLower() #input production environment name e.g. the "tmfgroup" in rg"tmfgroup" ***PIPELINE VAR***
$CompanyName = ($CompanyNametemp.subString(0, [System.Math]::Min(8, $CompanyNametemp.Length))).Trim()
$sgname = "sg${CompanyName}"
$rgname = "rg${CompanyName}"
$kvName = "kv${CompanyName}"
$sqlName = "sql${CompanyName}${envtype}"
*************************** Error**********************
InvalidOperation: /home/vsts/work/1/s/STAGE2/Scripts/firewallrules.ps1:4
Line |
4 | $CompanyNameTemp = $CompanyName.ToLower() #input production environme …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| You cannot call a method on a null-valued expression.
Agree with Krzysztof Madej.
As far as I know, linux system is case sensitive.
When you set the variable in azure devops, it will be converted to uppercase letters in environment variables.
For example:
So when you use the environment variable in Linux system, you need to change the format as
$env:COMPANY_NAME.
By the way, you could use the script to output all environment variables(e.g. env | sort).
Then you could check the environment variable format.