How to store Databricks token created from CLI in Yaml Bash step - azure-devops

I have the following Yaml script. I am looking for how to grab the token created and store into a variable:
- bash: |
echo {} > ~/.databricks-connect#
source py37-venv/bin/activate
pip3 install wheel
pip3 install databricks-cli
displayName: Install Databricks CLI
- bash: |
source py37-venv/bin/activate
databricks configure --token <<EOF
${DATABRICKS_HOST}
${DATABRICKS_AAD_TOKEN}
EOF
databricks tokens create --lifetime-seconds 129600 --comment "My comment."
The response that the above command returns is this json:
{
"token_value": "dapi1a23b45678901cd2e3fa4bcde56f7890",
"token_info": {
"token_id": "1ab23cd45678e90123f4567abc8d9e012345fa67890123b45678cde90fa123b4",
"creation_time": 1621287738473,
"expiry_time": 1621417338473,
"comment": "My comment."
}
}
I want to store the value of token_value above so I can use it in another task below.

You can use jq to parse the response json to get token value, for example:
token=$(databricks tokens create --lifetime-seconds 129600 --comment "My comment." | jq .token_value --raw-output)
Set $token as variable with logging command(you can set it as secret or not,click the link to check the usage), then use it in next job($(setvar.databrickstoken)).
echo "##vso[task.setvariable variable=databrickstoken;issecret=true;isoutput=true]$token"

Related

Github Workflow: Unable to process file command 'env' successfully

I'm using a github workflow to automate some actions for AWS. I haven't changed anything for a while as the script has been working nicely for me. Recently I've been getting this error: Unable to process file command 'env' successfully whenever the workflow runs. I've got no idea why this is happening. Any help or pointers would greatly appreciated. Thanks. Here's the workflow which is outputting the error:
- name: "Get AWS Resource values"
id: get_aws_resource_values
env:
SHARED_RESOURCES_ENV: ${{ github.event.inputs.shared_resources_workspace }}
run: |
BASTION_INSTANCE_ID=$(aws ec2 describe-instances \
--filters "Name=tag:env,Values=$SHARED_RESOURCES_ENV" \
--query "Reservations[*].Instances[*].InstanceId" \
--output text)
RDS_ENDPOINT=$(aws rds describe-db-instances \
--db-instance-identifier $SHARED_RESOURCES_ENV-rds \
--query "DBInstances[0].Endpoint.Address" \
--output text)
echo "rds_endpoint=$RDS_ENDPOINT" >> $GITHUB_ENV
echo "bastion_instance_id=$BASTION_INSTANCE_ID" >> $GITHUB_ENV
From the RDS endpoint query expression (Reservations[*].Instances[*].InstanceId) in your aws cli command, it seems you expect a multiline string. It could also be that before you started to receive this error the command was producing a single line string, and that changed at some point.
In GitHub actions, multiline strings for environment variables and outputs need to be created with a different syntax.
For the RDS endpoint you should set the environment variable like this:
echo "rds_endpoint<<EOF" >> $GITHUB_ENV
echo "$RDS_ENDPOINT" >> $GITHUB_ENV
echo "EOF" >> $GITHUB_ENV
I guess that the bastion instance id will not be a problem since it's a single line string.

How can I get Azure pipeline log file from command line

I have an Azure pipeline. I can start and see the logs in Chrome. But I would like to do these steps from command line (actually Cygwin, but IMHO this is not relevant).
az pipelines run --name $pipeline --branch $branch
This command gives back a json file format text on the stdout. This json has a entry, called logs:
{
...
"logs": {
"id": 0,
"type": "Container",
"url": "https://dev.azure.com/myazure/a56234f9-0101-5183-b422-db6f8cb55857/_apis/mybuild/mybuilds/1822266/logs"
},
...
}
If I copy the "url" into Chrome I get another json format homepage, like:
{"count":27,"value":[{"lineCount":371,"createdOn":"2022-10-17T13:38:14.013Z","lastChangedOn":"2022-10-17T13:38:14.183Z","id":1,"type":"Container","url":"https://dev.azure.com/myazure/...
But I cannot get back this json data from command line. I tried to get by curl or wget. I got back a HTML page (full with with JavaScript), not the json answer.
I also tried az:
az rest --method get --url "$logs_url"
But the response is:
Can't derive appropriate Azure AD resource from --url to acquire an access token. If access token is required, use --resource to specify the resource
Not a json response, outputting to stdout. For binary data suggest use "--output-file" to write to a file
Then I tried to do:
az account get-access-token --query accessToken --output tsv > access_token.tsv
az rest --method get --resource access_token.tsv --url "$logs_url"
So, I assume I should get an access token to the URL. But how can I acquire it?
You can use curl to call rest api: https://learn.microsoft.com/en-us/rest/api/azure/devops/pipelines/logs/list?view=azure-devops-rest-6.0
E.g.
curl https://dev.azure.com/{organization}/{project}/_apis/pipelines/{pipelineId}/runs/{runId}/logs?api-version=6.0-preview.1 ' -H 'Authorization: Basic YourPAT'

How to perform substring like function in jmespath on a string

I would like to know if there is substring function one can leverage in JMESPATH (supported by az cli).
I have the below az cli request and I want to just extract the name of the linked subnet with a security group, but unlike other cloud providers azure doesn't store associated resources names the same way.
The name can be extracted in the subnet.id node which looks like below
$ az network nsg show -g my_group -n My_NSG --query "subnets[].id" -o json
[
"/subscriptions/xxxxxx2/resourceGroups/my_group/providers/Microsoft.Network/virtualNetworks/MY-VNET/subnets/My_SUBNET"
]
I want to only extract "MY_SUBNET" from the the result.
I know there is something called search that is supposed to mimic
substring (explained here
https://github.com/jmespath/jmespath.jep/issues/5) but it didn't
work for me .
$ az network nsg show -g my_group -n My_NSG --query "subnets[].search(id,'#[120:-1]')" -o json
InvalidArgumentValueError: argument --query: invalid jmespath_type value: "subnets[].search(id,'#[120:-1]')"
CLIInternalError: The command failed with an unexpected error. Here is the traceback:
Unknown function: search()
Thank you
Edit :
I actually run the request including other elements that's why using substring with bash in a new line is not what I want .
here's an example of the full query :
az network nsg show -g "$rg_name" -n "$sg_name" --query "{Name:name,Combo_rule_Ports:to_string(securityRules[?direction==\`Inbound\`].destinationPortRanges[]),single_rule_Ports:to_string(securityRules[?direction==\`Inbound\`].destinationPortRange),sub:subnets[].id,resourceGroup:resourceGroup}" -o json
output
{
"Combo_rule_Ports": "[]",
"Name": "sg_Sub_demo_SSH",
"resourceGroup": "brokedba",
"single_rule_Ports": "[\"22\",\"80\",\"443\"]",
"sub": [
"/subscriptions/xxxxxxx/resourceGroups/brokedba/providers/Microsoft.Network/virtualNetworks/CLI-VNET/subnets/Sub_demo"
]
}
I had a similar problem with EventGrid subscriptions and used jq to transform JSON returned by the az command. As a result, you get an JSON array.
az eventgrid event-subscription list -l $location -g $resourceGroup --query "[].{
Name:name,
Container:deadLetterDestination.blobContainerName,
Account:deadLetterDestination.resourceId
}" \
| jq '[.[] | { Name, Container, Account: (.Account | capture("storageAccounts/(?<name>.+)").name) }]'
The expression Account: (.Account | capture("storageAccounts/(?<name>.+)").name) transforms the original resourceId from the Azure CLI.
# From Azure resourceId...
"Account": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/my-resource-group/providers/Microsoft.Storage/storageAccounts/mystorageaccount"
# .. to Azure Storage Account name
"Account": "mystorageaccount"
I've adapted the approach from How to extract a json value substring with jq.
cut can be used to extract desired values:
az network nsg show -g my_group -n My_NSG --query "subnets[].id|[0]" -o json | cut -d"/" -f11
If you run Azure CLI in bash, here are string manipulation operations you can do:
Following syntax deletes the longest match of $substring from the front of $string
${string##substring}
In this case, you can retrieve the subnet like this.
var=$(az network nsg show -g nsg-rg -n nsg-name --query "subnets[].id" -o tsv)
echo ${var##*/}
For more information, you could refer to https://www.thegeekstuff.com/2010/07/bash-string-manipulation/

Export ARM_CLIENT_ID and ARM_CLIENT_SECRET from Service connection in pipeline.yaml

I have created pipeline to import existing Azure Resource into terraform. Since Terraform Import requires Provider details or Environment Variables for The below details which has to extracted from the Service Connection.
steps:
- task: AzureCLI#2
displayName: Terraform Init
inputs:
azureSubscription: ${{ parameters.service_connection }}
addSpnToEnvironment: true
scriptType: bash
scriptLocation: inlineScript
inlineScript: |
export ARM_CLIENT_ID=$servicePrincipalId
export ARM_CLIENT_SECRET=$servicePrincipalKey
export ARM_SUBSCRIPTION_ID=$(az account show --query id | xargs)
export ARM_TENANT_ID=$(az account show --query tenantId | xargs)
ls
terraform init -upgrade -input=false \
-backend-config="subscription_id=${{ parameters.tf_state_subscription_id }}" \
-backend-config="tenant_id=$tenantId" \
-backend-config="client_id=$servicePrincipalId" \
-backend-config="client_secret=$servicePrincipalKey" \
-backend-config="resource_group_name=${{ parameters.resource_group_name }}" \
-backend-config="storage_account_name=${{ parameters.storage_account_name }}" \
-backend-config="container_name=${{ parameters.tf_state_key }}" \
-backend-config="key=${{ parameters.tf_state_key }}.tfstate"
if [ $(az resource list --name pytestkeyvault --query '[].id' -o tsv) != null ]
then
echo "using Keyvault $(az resource list --name pytestkeyvault --query '[].id' -o tsv)"
terraform import azurerm_key_vault.this $(az resource list --name pytestkeyvault --query '[].id' -o tsv)
else
echo "Keyvault does not exist"
fi
echo $ARM_CLIENT_ID
The exported environment variable ARM_CLIENT_ID is empty. The below variables are not being exported as environment variables.
echo $ARM_CLIENT_ID
echo $ARM_CLIENT_SECRET
echo $ARM_SUBSCRIPTION_ID
echo $ARM_TENANT_ID
For my setup i could not access the service principal from azure powershell.
But i could from Azure CLI.
This post pointed me in the right direction, check it out:
https://www.integration-playbook.io/docs/combining-az-cli-and-azure-powershell-az-modules-in-a-pipeline
In my experience of trying every possible variation of setting environment variables, it seems as ADO build agents don't allow the persisting of ARM_CLIENT_SECRET as an environment variable.
So the workaround I had to do was set the environment variables at the task level (instead of at the shell/machine level):
- script: |
terraform init # ...rest of your CLI arguments/backend-config flags
env:
ARM_CLIENT_SECRET: $(client_secret)
displayName: Terraform Init
Edit:
IMO, just using terraform init yourself via CLI is better than using the AzureCLI#2 task which is a confusing black box that honestly makes it harder/more verbose to do the same thing just with the plan CLI command.
Try using system variables $env:servicePrincipalId, $env:servicePrincipalKey, $env:tenantId to get SPN details.

Initialise and pull terraform public modules using GitHub SSH private key

Context:
I have gitlab runners which are executing terraform init command which is pulling all necessary terraform modules. Recently, I started hitting github throttling issues (60 calls to github api per hour). So I am trying to reconfigure my pipeline so it uses Github user's private key.
Currently, I have the following in my pipeline but it still doesn't seem to work and private key isn't used to pull the terraform modules.
- GITHUB_SECRET=$(aws --region ${REGION} ssm get-parameters-by-path --path /github/umotifdev --with-decryption --query 'Parameters[*].{Name:Name,Value:Value}' --output json);
- PRIVATE_KEY=$(echo "${GITHUB_SECRET}" | jq -r '.[] | select(.Name == "/github/umotifdev/private_key").Value' | base64 -d);
- PUBLIC_KEY=$(echo "${GITHUB_SECRET}" | jq -r '.[] | select(.Name == "/github/umotifdev/public_key").Value' | base64 -d);
- mkdir -p ~/.ssh;
- echo "${PRIVATE_KEY}" | tr -d '\r' > ~/.ssh/id_rsa;
- chmod 700 ~/.ssh/id_rsa;
- eval $(ssh-agent -s);
- ssh-add ~/.ssh/id_rsa;
- ssh-keyscan -H 'github.com' >> ~/.ssh/known_hosts;
- ssh-keyscan github.com | sort -u - ~/.ssh/known_hosts -o ~/.ssh/known_host;
- echo -e "Host github.com\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config;
- echo ${PUBLIC_KEY} >> ~/.ssh/authorized_keys
The error I am seeing in my pipeline is something like (which is basically throttling from github):
Error: Failed to download module
Could not download module "vpc" (vpc.tf:17) source code from
"https://api.github.com/repos/terraform-aws-modules/terraform-aws-vpc/tarball/v2.21.0//*?archive=tar.gz":
bad response code: 403.
Anyone can advise how to resolve an issue where private key isn't used to pull terraform modules?