I have recently started with ARM and Azure Data Factory, and now I have faced a problem when I try to deploy ADF into other environments. Because the source connection string are different in each environment and I can not keep the static value in adf.content.parameters.json file.
So I have created three YML files for each environment as Dev.yml, Test.yml, and Prod.yml. I have three files
adf.content.json
adf.content.parameters.json
Dev.yml, test.yml and prod.yml
In the file, adf.content.json, I have a connection string as my source. This value is changing in each environment. Here is the adf.content.parameters.json
"parameters": {
"source_connectionString": {
"type": "secureString",
"metadata": "Secure string for 'connectionString' of 'source-db'"
},
and I removed this parameter from adf.content.parameters.json but instead, I have added that to Dev.yml file which looks like this one (Test.yml and prod.yml are the same just different values)
variables:
- name: source-connectionstring
value: <some value>
I have a ci-build.yml and ci-deploy.yml files which will be used for the CI pipeline. In the ci-build.yml file, I use the same name in the adf.content.json
variables:
SourceConnectionString: $(source_connectionString:?You need to set the source_connectionString environment variable)
- stage: DevDeploy
displayName: Deploy to development (D)
variables:
- template: ../.../.../dev.yml
- group: ...
dependsOn: Build
jobs:
- template: cd_deploy.yml
parameters:
environment: dev
azureServiceConnection: '...'
containerRegistryDomain: ...
apiResourceGroup: '...'
webAppName: '...
azureSubscriptionName: ...
apimResourceGroup: ...
apimName: ...
And in my ci-deply.yml file looks like this
task: AzureResourceGroupDeployment#2
displayName: "Deploy Azure Data Factory Content"
inputs:
azureSubscription: '...'
action: '...'
resourceGroupName: '...'
location: '...'
templateLocation: '...'
csmFile: '$(System.ArtifactsDirectory)/.../arm/adf.content.json'
csmParametersFile: '$(System.ArtifactsDirectory)/../arm/adf.content.parameters.json'
overrideParameters: ' -source_connectionString$(SourceConnectionString)
deploymentMode: 'Incremental'
But I get an error " did not find expected key" on overrideParameters: ' -source_connectionString$(SourceConnectionString)
I do not know if this is the right approach for this? and if anyone can get why I can not get the key?
This error is usually caused by wrong syntax.
In ci-deply.yml, the script overrideParameters: ' -source_connectionString$(SourceConnectionString) loses a '.
So your task in ci-deply.yml should look like this:
- task: AzureResourceGroupDeployment#2
displayName: "Deploy Azure Data Factory Content"
inputs:
azureSubscription: '...'
action: '...'
resourceGroupName: '...'
location: '...'
templateLocation: '...'
csmFile: '$(System.ArtifactsDirectory)/.../arm/adf.content.json'
csmParametersFile: '$(System.ArtifactsDirectory)/../arm/adf.content.parameters.json'
overrideParameters: ' -source_connectionString$(SourceConnectionString) '
deploymentMode: 'Incremental'
Related
I have a main.bicep template with multiple resources that I deploy to Azure via my DevOps pipeline. My template is currently configured to either deploy all modules to Azure or none. However, I would like to customize this in my pipeline so that only certain resources show up in my resource group so that it acts like a selection catalog.
However, I need advice and help on how to solve the problem as I can’t find the right approach. Attached you can see a snippet from my bicep template below.
– main.bicep–
param deployAppInsights bool = true param deployNetworkWatcher bool = true param deploySentinel bool = true
Example for a module
// Deploy App Insights resource
module appInsights 'AzureMonitor/AppInsights/appInsights-temp.bicep' = if(deployAppInsights) { name: 'appInsightsDeployment' params: { appName: appInsightsName regionId: regionId // tagsArray: tagsArray requestSource: requestSource workspaceResourceId: workspace.outputs.resourceWorkspaceIdOutput } }
This snippet is in my pipeline - yml
(I would like to shorten the parameters block by using override parameters.)
parameters:
- name: appInsights
type: boolean
default: True
values:
- True
- False
inputs: overrideParameters: -deployAppInsights false -deployNetworkWatcher true -deploySentinel true
Currently, my override parameters are hardcoded, but I’d like to keep it in a general way, like for example:
overrideParameters: -deployAppInsights ${{appInsights}} -deployNetworkWatcher ${{networkWatcher}} -deploySentinel ${{sentinel}}
However, this does not unfortunately work. Could anyone please help me in this regard, how I can solve this or how the variables need to look like?
Thank you very much!
Best regards
Jennifer
As far as I understand, the syntax you are using is wrong in the below code
overrideParameters: -deployAppInsights ${{appInsights}} -deployNetworkWatcher ${{networkWatcher}} -deploySentinel ${{sentinel}}
Since you have already defined parameters section in the Yml file. The correct syntax is
overrideParameters: -deployAppInsights ${{parameters.appInsights}} -deployNetworkWatcher ${{parameters.networkWatcher}} -deploySentinel ${{parameters.sentinel}}
A sample code snippet for your reference
parameters:
- name: var1
type: string
- name: var2
type: string
- name: bool1
type: bool
...
omitted
...
- task: AzureResourceManagerTemplateDeployment#3
inputs:
deploymentScope: 'Resource Group'
azureResourceManagerConnection: ''
subscriptionId: ''
action: 'Create Or Update Resource Group'
resourceGroupName: rg-${{parameters.var1}}-${{parameters.var2}}
location: ''
templateLocation: 'Linked artifact'
csmFile: $(System.DefaultWorkingDirectory)/arm.json
csmParametersFile: $(System.DefaultWorkingDirectory)/arm.parameters.json
deploymentMode: 'Incremental'
overrideParameters:
-string1 ${{parameters.var1}}
-string2 ${{parameters.var2}}
-flag ${{parameters.bool1}}
For more, please refer to the link from Microsoft. Azure Devops Template
Edit: After the first comment
Please update your 'appInsights' parameter as follows
parameters:
- name: appInsights
type: boolean
default: true
I am trying to pass to a python scripts 3 parameters one of them is an array, this works when i run the script locally, i am using sys.argv to achieve this.
However the argument field support only strings as far i can see. How can i go around this any ideas? Thanks
the array is ${{ parameters.packageVersion }}
Code:
- task: PythonScript#0
displayName: 'Modify ansible inventory files (wms_common.yml) with deployed versions'
inputs:
scriptSource: filePath
scriptPath: deployment/s/azure-devops/scripts/script.py
arguments: |
../../inventories/${{ variables.inventory }}/group_vars/all/wms_common.yml
../../inventories/central/${{ variables.inventory }}/group_vars/all/wms_common.yml
${{ parameters.packageVersion }}
Error:
/azure-devops/wms.full.pipeline.yml (Line: 95, Col: 18): Unable to convert from Array to String. Value: Array
Edit: Reframed question
I think the below YAML will help you convert the array to String objects and use them.
variables:
myVariable1: 'value1'
myVariable2: 'value2'
system.debug: true
parameters:
- name: InstanceArgs
type: object
default: [1,2]
steps:
- task: PythonScript#0
inputs:
scriptSource: 'inline'
script: |
import argparse
parse = argparse.ArgumentParser(description="Test")
parse.add_argument("test1")
parse.add_argument("test2")
args = parse.parse_args()
print(args.test1)
print(args.test2)
print('this is a test.')
# arguments: $(myVariable1) $(myVariable2)'
arguments: ${{join(' ',parameters.InstanceArgs)}}
You need to use the join expression in YAML to get the elements:
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/expressions?view=azure-devops#join
I have the following scenario, it is simplified for the sake of brevity, but outlines my problem.
I have a 2 Job pipeline.
BreakMatrix Job: A job that runs on an AdminPool and outputs 2 variables with the following names ENV1 and ENV2. The names are important because each of them matches the name of an Environment running in a separate MachinePool VM deployment pool.
Upgrade Job: A deployment job that depends on the BreakMatrix job and that runs on a VM MachinePool, with a tag that will select ENV1 and ENV2 environments.
I am trying to pass in one of the variables to each of the corresponding Environments:
- job: BreakMatrix
pool: AdminPool
steps:
- checkout: none
- powershell: |
$result1 = #{"Hostname" = "Env1Value"}
$result2 = #{"Hostname" = "Env2Value"}
Write-Host "##vso[task.setvariable variable=ENV1;isOutput=true;]$result1"
Write-Host "##vso[task.setvariable variable=ENV2;isOutput=true;]$result2"
name: outputter
- deployment: Upgrade
dependsOn: BreakMatrix
variables:
agentName: $(Environment.ResourceName)
agentName2: $(Agent.Name)
formatted: $[ format('outputter.{0}', variables['agentName']) ]
result1: $[ dependencies.BreakMatrix.outputs[format('outputter.{0}', variables['agentName'])] ]
result2: $[ dependencies.BreakMatrix.outputs[format('outputter.{0}', variables['agentName2'])] ]
result3: $[ dependencies.BreakMatrix.outputs[format('outputter.{0}', variables['Agent.Name'])] ]
hardcode: $[ dependencies.BreakMatrix.outputs['outputter.ENV2'] ]
json: $[ convertToJson(dependencies.BreakMatrix.outputs) ]
environment:
name: MachinePool
resourceType: VirtualMachine
tags: deploy-dynamic
strategy:
rolling:
preDeploy:
steps:
- powershell: |
echo 'Predeploy'
echo "agentName: $(agentName)"
echo "agentName2: $(agentName2)"
echo "env: $(Environment.ResourceName)"
echo "formatted: $(formatted)"
echo "harcode: $(harcode)"
echo "result1: $(result1)"
echo "result2: $(result2)"
echo "result3: $(result3)"
echo "json: $(json)"
deploy:
steps:
- powershell: |
echo 'Deploy'
Output for ENV2 pre-deploy step:
Predeploy
agentName: ENV2
agentName2: ENV2
env: ENV2
formatted: outputter.ENV2
hardcode: {
Hostname:Env2Value}
result1:
result2:
result3:
json: {
outputter.ENV2:
\"Hostname\":\"Env2Value\"
}
If i try to use a predefined variable in a dependency expression they don't seem to properly resolve, but if i just simply map them to a variable / format them they work.
Note: The environment names are actually dynamically calculated before these jobs run so I cannot use parameters or static/compile time variables.
Any suggestions on how to pass in only the relevant variable to each of the environments ?
I got confirmation on the developer community that this is not possible.
Relevant part:
But I am afraid that the requirement couldn’t be achieved.
The reason is that the dependencies.xx.outputs expression can’t read
the variable(variables['Agent.Name']) and format expression value
defined in the YAML pipeline.
It now only supports hard-coding the name of the variable to get the
corresponding job variable value.
In my build pipeline I've defined a variable that holds a password:
To reference a variable in YAML, prefix it with a dollar sign and enclose it in parentheses. For example: $(ACCOUNT_PASSWORD)
I have a task that uses the file creator extension:
- task: file-creator#6
inputs:
filepath: '$(System.DefaultWorkingDirectory)/cypress.env.json'
filecontent: |
{
"ACCOUNT_PASSWORD": $(ACCOUNT_PASSWORD)
}
fileoverwrite: true
But I get this error:
SyntaxError: /home/**/_work/1/s/cypress.env.json: Unexpected token $ in JSON at position 24
So I create a json file and I think the $(ACCOUNT_PASSWORD) is interpreted as a regular string in stead of the value of the variable.
Not sure what kind of syntax, it is expecting but you can try variables['ACCOUNT_PASSWORD']
This worked for me:
- task: file-creator#6
inputs:
filepath: '$(System.DefaultWorkingDirectory)/cypress.env.json'
filecontent: |
{
"ACCOUNT_PASSWORD": "ACCOUNT_PASSWORD"
}
fileoverwrite: true
- task: FileTransform#1
inputs:
folderPath: '$(System.DefaultWorkingDirectory)'
fileType: 'json'
targetFiles: 'cypress.env.json'
Azure ARM Template parameter.json file has the below object property.
"appInsightsObject": {
"value": {
"name": "appInsghtName",
"id": "appInsgID"
}
}
I have to replace these values from build.yaml file. My build.yaml file has the below
- task: AzureResourceManagerTemplateDeployment#3
displayName: APIM Development CI
inputs:
ConnectedServiceName: My-Service-Name
subscriptionName: My-Subs-Values
resourceGroupName: My-Rg-Name
location:$(locationName)
csmFile: template.json
csmParametersFile: parameters.json
overrideParameters: '-appInsightsObject.name $(appInsightNameValue) '
How to pass appInsightsObject object value?
Update:
I found one way of passing the value as JSON object like '{"name":"name-goes-here", "id":"id-value-goes-here"}'. Is there any better option?
overrideParameters: '-appInsightsObject $(appInsightValue) '
This one works and allows to store values individually for any type of object.
overrideParameters: '-appInsightsObject {"name": "$(dev.insightName)","id": "$(dev.insightId)"} '