I am trying to configure a devops pipeline to get Databricks workspace to automatically pull the latest version of the repo when master branch is updated.
I am able to generate the DB_PAT but I cannot seem to get this into the DATABRICKS_TOKEN variable properly hence the databricks repos update fails with the following error:
Error: JSONDecodeError: Expecting value: line 1 column 1 (char 0)
##[debug]Exit code 1 received from tool '/usr/bin/bash'
##[debug]STDIO streams have closed for tool '/usr/bin/bash'
##[error]Bash exited with code '1'.
This is my YAML file that I ran in Azure Devops.
trigger:
- master
jobs:
- job: PullReposUpdate
steps:
- task: AzureCLI#2
name: DbTokenGen
inputs:
azureSubscription: $(azureSubscription)
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
ADBWORKSPACENAME="adb-$(clientName)-$(stageName)"
echo "$ADBWORKSPACENAME"
TENANT_ID=$(az account show --query TENANT_ID --output tsv)
WORKSPACE_ID=$(az resource show --resource-type Microsoft.Databricks/workspaces --resource-group $RGNAME --name $ADBWORKSPACENAME --query id --output tsv)
TOKEN=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d | jq --raw-output '.accessToken')
AZ_TOKEN=$(az account get-access-token --resource https://management.core.windows.net/ | jq --raw-output '.accessToken')
DB_PAT=$(curl --silent https://$(region).azuredatabricks.net/api/2.0/token/create \
--header "Authorization: Bearer $TOKEN" \
--header "X-Databricks-Azure-SP-Management-Token:$AZ_TOKEN" \
--header "X-Databricks-Azure-Workspace-Resource-Id:$WORKSPACE_ID" \
--data '{ "lifetime_seconds": 1200, "comment": "Azure DevOps pipeline" }' \
| jq --raw-output '.token_value')
echo "##vso[task.setvariable variable=DB_PAT]$DB_PAT"
failOnStandardError: true
displayName: 'Generate Token for Databricks'
- script: |
pip install --upgrade databricks-cli
displayName: 'Install dependencies'
- script: |
databricks repos update --path $(STAGING_DIRECTORY) --branch "$(branchName)"
env:
DATABRICKS_HOST: $(databricks-url)
DATABRICKS_TOKEN: $(DbTokenGen.DB_PAT)
displayName: 'Update Production Databricks Repo'
How can I pass the DB_PAT successfully to the databricks cli environment? any suggestion is highly appreciated.
Thank you.
Related
I've been trying to set policy for managed service identities for my key vault, but I haven't been able to create needed variable in my yml. According to the pipeline output the variable 'indentity' in the below code gets populated, but the variable WEB_APP_MANAGED_IDENTITY doesn't. Thus the last task "Add set policy for function app" naturally fails. What can I be doing incorrectly ?
parameters:
serviceConnector: ''
appResourceGroupName: ''
appName: ''
appPlan: ''
appInsightsName: ''
location: ''
planSkuName: ''
PostgresServerName: ''
AdminUserName: ''
AdminPassWord: ''
pgSkuName: ''
subscriptionId: ''
keyVaultName: ''
devOpsServicePrincipalObjectId: ''
verkkoGroupServicePrincipalObjectId: ''
stages:
- stage: 'CREATE_SERVICE_APP'
displayName: 'Create Osakeekstra service app'
jobs:
- job:
steps:
- checkout: none
- task: AzureCLI#2
displayName: 'Create and configure web app'
inputs:
azureSubscription: '$(serviceConnector)'
scriptLocation: 'inlineScript'
scriptType: 'bash'
failOnStandardError: false
inlineScript: |
echo ">>>> Create app service plan"
az appservice plan create \
--name $(appPlan) \
--resource-group $(appResourceGroupName) \
--location $(location) \
--sku $(planSkuName) \
--is-linux 2>/dev/null
# create web app
echo ">>>> Create web app"
az webapp create \
--resource-group $(appResourceGroupName) \
--name $(appName) \
--runtime "Python|3.7" \
--plan $(appPlan)
# enable managed identity
echo ">>>> Enable managed identity"
identity=`az webapp identity assign \
--name $(appName) \
--resource-group $(appResourceGroupName) \
--query principalId -o tsv`
echo "##vso[task.setvariable variable=WEB_APP_MANAGED_IDENTITY]$identity"
echo $WEB_APP_MANAGED_IDENTITY
echo $identity
echo ">>>> Enabled managed identity"
- task: AzureCLI#2
displayName: 'Create key vault'
inputs:
scriptType: bash
azureSubscription: '$(serviceConnector)'
scriptLocation: 'inlineScript'
inlineScript: |
echo ">>>> Create key vault"
az keyvault create \
--name $(keyVaultName) \
--resource-group $(appResourceGroupName) \
--location $(location) \
--no-self-perms \
--sku standard \
--enable-soft-delete true \
2>&1
# add set policy for managed service identities
echo ">>>> Add set policy for function app"
az keyvault set-policy \
--name $(keyVaultName) \
--secret-permissions get \
--object-id $(FUNCTION_APP_MANAGED_IDENTITY)
failOnStandardError: true
Whwn I changed the last command to this
# enable managed identity
echo ">>>> Enable managed identity"
identity=`az webapp identity assign \
--name $(appName) \
--resource-group $(appResourceGroupName) \
--query principalId -o tsv`
# add set policy for managed service identities
echo ">>>> Add set policy for function app"
az keyvault set-policy \
--name $(keyVaultName) \
--secret-permissions get \
--object-id $(indentity)
I got errors : indentity: command not found
ERROR: argument --object-id: expected one argument
There are a few issues here that I can see.
First, if memory serves, az webapp identity assign will return more then just the principalId, so you'll need to parse apart the result to get the correct value.
Second, to reference identity as a release variable (using macro syntax $(variablename)) instead of an environment variable, you need to add ;isOutput=true inside the task.setvariable statement, as in echo "##vso[task.setvariable variable=identity;isOutput=true]$identity" - so you're setting the release variable named identity to the value of then environment variable named identity.
Third, in your reference to identity in the following task, identity is misspelled as indentity.
Fourth, and this is least important, you do echo $WEB_APP_MANAGED_IDENTITY after the task.setvariable command - it's not available then - task.setvariable is a "marker" that Azure Pipelines uses to scrape the console output, and then use to set variables for subsequent tasks and jobs.
You can either set the variable as ##vso[task.setvariable variable=identity;isOutput=true]$identity and refer to it later as $(identity), or, using your original name, set the variable as echo "##vso[task.setvariable variable=WEB_APP_MANAGED_IDENTITY;isOutput=true]$identity" and refer to it later as $(WEB_APP_MANAGED_IDENTITY)
I am trying to use terraform to build the infra in Azure. I am automating it through Azure DevOps, we don't have the tasks in our org yet, I am running through CLI scripts to get it, while I am able to till terraform init but unable to run through terraform plan. I am using service principal to authenticate as mentioned here. I am following this to complete the setup.
Here is my pipeline.
- task: AzureCLI#1
displayName: Terraform init
inputs:
azureSubscription: Subscription
scriptLocation: inlineScript
inlineScript: |
set -eux # fail on error
terraform init \
-backend-config=storage_account_name=$(storageAccountName) \
-backend-config=container_name=$(container_name) \
-backend-config=key=$(key)/terraform.tfstate \
-backend-config=sas_token=$(artifactsLocationSasToken) \
-backend-config=subscription_id="$(ARM_SUBSCRIPTION_ID)" \
-backend-config=tenant_id="$(ARM_TENANT_ID)" \
-backend-config=client_id="$(ARM_CLIENT_ID)" \
-backend-config=client_secret="$(ARM_CLIENT_SECRET)"
addSpnToEnvironment: true
workingDirectory: $(System.DefaultWorkingDirectory)/Modules
- bash: |
set -eu # fail on error
terraform plan -out=tfplan -input=false -detailed-exitcode
displayName: Terraform apply
workingDirectory: $(System.DefaultWorkingDirectory)/Modules
and in the tf file I have very basic to try
provider "azurerm" {
version = ">= 2.61.0"
features {}
}
data "azurerm_resource_group" "main" {
name = var.resource_group_name
}
terraform {
backend "azurerm" {
}
}
I am getting this error.
Error building AzureRM Client: obtain subscription(***) from Azure
CLI: Error parsing json result from the Azure CLI: Error waiting for
the Azure CLI: exit status 1: ERROR: Please run 'az login' to setup
account.
Updated
- task: AzureCLI#2
inputs:
azureSubscription: $(scConn)
scriptType: 'pscore'
scriptLocation: 'inlineScript'
inlineScript: |
$sasToken = (az storage container generate-sas --account-name $(storageAccountName) --name $(container_name) --permissions rwdl --expiry $(date -u -d "30 minutes" +%Y-%m-%dT%H:%MZ))
Write-Host($sasToken) Write-Output("##vso[task.setvariable variable=artifactsLocationSasToken;]$sasToken")
- task: AzureCLI#1
displayName: Terraform credentials
inputs:
azureSubscription: $(scConn)
scriptLocation: inlineScript
inlineScript: |
set -eu # fail on error
echo "##vso[task.setvariable variable=ARM_CLIENT_ID]$(servicePrincipalId)"
echo "##vso[task.setvariable variable=ARM_CLIENT_SECRET;issecret=true]$(servicePrincipalKey)"
echo "##vso[task.setvariable variable=ARM_SUBSCRIPTION_ID]$(subscriptionId)"
echo "##vso[task.setvariable variable=ARM_TENANT_ID]$(tenantId)"
addSpnToEnvironment: true
- task: AzureCLI#1
displayName: Terraform init
inputs:
azureSubscription: $(scConn)
scriptLocation: inlineScript
inlineScript: |
set -eux # fail on error
terraform init \
-backend-config=storage_account_name=$(storageAccountName) \
-backend-config=container_name=$(container_name) \
-backend-config=key=$(key)/terraform.tfstate \
-backend-config=sas_token=$(artifactsLocationSasToken) \
-backend-config=subscription_id="$(ARM_SUBSCRIPTION_ID)" \
-backend-config=tenant_id="$(ARM_TENANT_ID)" \
-backend-config=client_id="$(ARM_CLIENT_ID)" \
-backend-config=client_secret="$(ARM_CLIENT_SECRET)"
addSpnToEnvironment: true
workingDirectory: $(System.DefaultWorkingDirectory)/Modules
Error building AzureRM Client: obtain subscription(***) from Azure CLI: Error parsing json result from the Azure CLI: Error waiting for the Azure CLI: exit status 1: ERROR: Please run 'az login' to setup account.
The root cause of this issue is that Azure CLI will run the az account clear command at the end. So the az login information in the current Azure CLI task will not be retained.
You need to add additional command (az login command)to login before the terraform plan command.
You could enable the parameter in Azure CLI Task: addSpnToEnvironment: true and set the login info as Pipeline variable. Then you could use these info in az login command.
Here is an example:
- task: AzureCLI#1
displayName: Terraform init
inputs:
azureSubscription: Subscription
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
set -eux # fail on error
echo "##vso[task.setvariable variable=ARM_CLIENT_ID]$servicePrincipalId"
echo "##vso[task.setvariable variable=ARM_CLIENT_SECRET]$servicePrincipalKey"
echo "##vso[task.setvariable variable=ARM_TENANT_ID]$tenantId"
terraform init \
-backend-config=storage_account_name=$(storageAccountName) \
-backend-config=container_name=$(container_name) \
-backend-config=key=$(key)/terraform.tfstate \
-backend-config=sas_token=$(artifactsLocationSasToken) \
-backend-config=subscription_id="$(ARM_SUBSCRIPTION_ID)" \
-backend-config=tenant_id="$(ARM_TENANT_ID)" \
-backend-config=client_id="$(ARM_CLIENT_ID)" \
-backend-config=client_secret="$(ARM_CLIENT_SECRET)"
addSpnToEnvironment: true
workingDirectory: $(System.DefaultWorkingDirectory)/Modules
- bash: |
set -eu # fail on error
az login --service-principal --username $(ARM_CLIENT_ID) --password $(ARM_CLIENT_SECRET) --tenant $(ARM_TENANT_ID)
terraform plan -out=tfplan -input=false -detailed-exitcode
displayName: Terraform apply
workingDirectory: $(System.DefaultWorkingDirectory)/Modules
I have created pipeline to import existing Azure Resource into terraform. Since Terraform Import requires Provider details or Environment Variables for The below details which has to extracted from the Service Connection.
steps:
- task: AzureCLI#2
displayName: Terraform Init
inputs:
azureSubscription: ${{ parameters.service_connection }}
addSpnToEnvironment: true
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
export ARM_CLIENT_ID=$servicePrincipalId
export ARM_CLIENT_SECRET=$servicePrincipalKey
export ARM_SUBSCRIPTION_ID=$(az account show --query id | xargs)
export ARM_TENANT_ID=$(az account show --query tenantId | xargs)
ls
terraform init -upgrade -input=false \
-backend-config="subscription_id=${{ parameters.tf_state_subscription_id }}" \
-backend-config="tenant_id=$tenantId" \
-backend-config="client_id=$servicePrincipalId" \
-backend-config="client_secret=$servicePrincipalKey" \
-backend-config="resource_group_name=${{ parameters.resource_group_name }}" \
-backend-config="storage_account_name=${{ parameters.storage_account_name }}" \
-backend-config="container_name=${{ parameters.tf_state_key }}" \
-backend-config="key=${{ parameters.tf_state_key }}.tfstate"
if [ $(az resource list --name pytestkeyvault --query '[].id' -o tsv) != null ]
then
echo "using Keyvault $(az resource list --name pytestkeyvault --query '[].id' -o tsv)"
terraform import azurerm_key_vault.this $(az resource list --name pytestkeyvault --query '[].id' -o tsv)
else
echo "Keyvault does not exist"
fi
echo $ARM_CLIENT_ID
The exported environment variable ARM_CLIENT_ID is empty. The below variables are not being exported as environment variables.
echo $ARM_CLIENT_ID
echo $ARM_CLIENT_SECRET
echo $ARM_SUBSCRIPTION_ID
echo $ARM_TENANT_ID
For my setup i could not access the service principal from azure powershell.
But i could from Azure CLI.
This post pointed me in the right direction, check it out:
https://www.integration-playbook.io/docs/combining-az-cli-and-azure-powershell-az-modules-in-a-pipeline
In my experience of trying every possible variation of setting environment variables, it seems as ADO build agents don't allow the persisting of ARM_CLIENT_SECRET as an environment variable.
So the workaround I had to do was set the environment variables at the task level (instead of at the shell/machine level):
- script: |
terraform init # ...rest of your CLI arguments/backend-config flags
env:
ARM_CLIENT_SECRET: $(client_secret)
displayName: Terraform Init
Edit:
IMO, just using terraform init yourself via CLI is better than using the AzureCLI#2 task which is a confusing black box that honestly makes it harder/more verbose to do the same thing just with the plan CLI command.
Try using system variables $env:servicePrincipalId, $env:servicePrincipalKey, $env:tenantId to get SPN details.
I have creates below task to find the current tag and pass on to next tasks while building the docker image with new tag.
- task: Bash#3
displayName: 'Fetch latest tag from ECR and Create Next Tag.'
inputs:
targetType: 'inline'
script: |
ecrURI=$(ecrURI)
repoName="${ecrURI##*/}"
latestECRTag=$(aws ecr describe-images --output json --repository-name ${repoName} --region $(awsDefaultRegion) --query 'sort_by(imageDetails,& imagePushedAt)[-1].imageTags[0]' | jq . --raw-output)
if [[ -z ${latestECRTag} ]];
then
latestECRTag='0.0.0'
fi
major=$(echo ${latestECRTag} |cut -d'.' -f1)
minor=$(echo ${latestECRTag} |cut -d'.' -f2)
patch=$(echo ${latestECRTag} |cut -d'.' -f3)
latestECRTag="$(expr $major + 1).${minor}.${patch}"
echo $latestECRTag
echo "##vso[task.setvariable variable=NEXT_ECR_TAG;isOutput=true]$latestECRtag"
- bash: |
echo "Started Building Docker Image with tag $latestECRTag"
docker build -t test:latest -f Dockerfile .
docker tag test:latest $(ecrURI):$(NEXT_ECR_TAG)
displayName: 'Build Docker Image with Tag.'
workingDirectory: $(System.DefaultWorkingDirectory)/configCreate/
The step/task to fetch and create the new tag is working fine, but as soon as I moved to the next task to build the docker tag based on NEXT_ECR_TAG it shows me empty value. everything else are properly populated.
Can anyone help me to find out why i'm not able to fetch the NEXT_ECR_TAG value in next task? It could be silly thing, but don't know what?
duh!! moment for me. there is a typo in my variable. Fixing typo fixed everything.
I've successfully run pipeline with step below to push docker image to registry. When I try to re-run it gives an ERROR: Please run 'az login' to setup account
- bash: az acr helm push -n $(registryName) -u $(registryLogin) -p $(registryPassword) $(build.artifactStagingDirectory)/$(projectName)-$(build.buildId).tgz
displayName: 'az acr helm push'
condition: and(succeeded(), eq(variables['build.sourceBranch'], 'refs/heads/master'))
The step is seen in the logs as below so shouldn't require any additional information (it works when run from command line on my local machine)
az acr helm push -n acrname -u acruser -p password /home/vsts/work/1/a/chart.tgz
The azure cli genuinely does need you to be logged in, so az cli commands in a bash task will fail with this error. That is expected behaviour.
Instead of using the bash task, you should use the Azure CLI task; this includes authentication against an azure subscription as part of its setup, so you will be able to run a bash script including az cli commands.
For example:
- task: AzureCLI#2
displayName: az acr helm push
inputs:
azureSubscription: <Name of the Azure Resource Manager service connection>
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
az acr helm push -n $(registryName) -u $(registryLogin) -p $(registryPassword) $(build.artifactStagingDirectory)/$(projectName)-$(build.buildId).tgz