I'm attempting to publish our development ADF objects from our development subscription to our test subscription.
To accomplish this, I used the Microsoft guide to set up a release pipeline. My final result looks like this:
However, when attempting to deploy to test, it fails:
2022-05-04T12:22:05.6519328Z ##[section]Starting: ARM Template deployment: Resource Group scope 2022-05-04T12:22:05.6923741Z
============================================================================== 2022-05-04T12:22:05.6925480Z Task : ARM template deployment 2022-05-04T12:22:05.6925967Z Description : Deploy an Azure Resource Manager (ARM) template to all the deployment scopes 2022-05-04T12:22:05.6926257Z Version : 3.200.0 2022-05-04T12:22:05.6926589Z Author : Microsoft Corporation 2022-05-04T12:22:05.6927048Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment 2022-05-04T12:22:05.6927663Z
============================================================================== 2022-05-04T12:22:06.5933183Z ARM Service Connection deployment scope - Subscription 2022-05-04T12:22:06.6193437Z Checking if the following resource group exists: tst-<Removed>. 2022-05-04T12:22:06.8901626Z Resource group exists: true. 2022-05-04T12:22:06.8904567Z Creating deployment parameters. 2022-05-04T12:22:06.9013317Z The detected encoding for file 'D:\a\r1\a\Dev\dev-adf-<Removed>\ARMTemplateParametersForFactory.json' is 'utf-8' 2022-05-04T12:22:06.9064165Z (node:4736) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead. 2022-05-04T12:22:07.0403949Z Starting template validation. 2022-05-04T12:22:07.0478298Z Deployment name is ARMTemplateParametersForFactory-20220504-122207-af92 2022-05-04T12:22:07.3412253Z There were errors in your deployment. Error code: InvalidRequestContent. 2022-05-04T12:22:07.3458400Z
##[error]The request content was invalid and could not be deserialized: 'Required property 'type' not found in JSON. Path 'properties.template.parameters.factoryName', line 1, position 208.'. 2022-05-04T12:22:07.3475499Z ##[warning]Validation errors were found in the Azure Resource Manager template. This can potentially cause template deployment to fail. Task failed while creating or updating the template deployment.. Please follow https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/template-syntax 2022-05-04T12:22:07.3477016Z Starting Deployment. 2022-05-04T12:22:07.3477522Z Deployment name is ARMTemplateParametersForFactory-20220504-122207-af92 2022-05-04T12:22:07.4293852Z There were errors in your deployment. Error code: InvalidRequestContent. 2022-05-04T12:22:07.4295964Z
##[error]The request content was invalid and could not be deserialized: 'Required property 'type' not found in JSON. Path 'properties.template.parameters.factoryName', line 1, position 208.'. 2022-05-04T12:22:07.4298521Z ##[error]Check out the troubleshooting guide to see if your issue is addressed: https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment?view=azure-devops#troubleshooting 2022-05-04T12:22:07.4301572Z ##[error]Task failed while creating or updating the template deployment. 2022-05-04T12:22:07.4465058Z
##[section]Finishing: ARM Template deployment: Resource Group scope
The contents of the ARMTemplateParametersForFactory.json:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"factoryName": {
"value": "dev-adf-<removed>"
},
"<removed>_EDW_connectionString": {
"value": ""
},
"dev_blob_<removed>_connectionString": {
"value": ""
}
}
}
As you can see, there is no line 208 in that file. The logs are not helpful at all, and to the best of my knowledge, I have not made any changes to the parameter file.
I have tried reading related questions, but they all seem to deal with different issues or are not related to Data Factory projects. Checking other files within the adf_publish branch, I did see some of them had a "type":"string" property. I attempted to manually add that to the ArmTemplateParametersForDataFactory file as well, but the error remains.
I'm franmkly not sure how to proceed. Any help would be appreciated. Thank you.
Related
I have a theme with a size >1MB (which precludes the configmap-solution provided as an answer to this question).
This theme has been been packaged according to the Server Development Guide - its folder structure is
META-INF/keycloak-themes.json
themes/[themeName]/login/login.ftl
themes/[themeName]/login/login-reset-password.ftl
themes/[themeName]/login/template.ftl
themes/[themeName]/login/template.html
themes/[themeName]/login/theme.properties
themes/[themeName]/login/messages/messages_de.properties
themes/[themeName]/login/messages/messages_en.properties
themes/[themeName]/login/resources/[...]
The contents of keycloak-themes.json are
{
"themes": [{
"name" : "[themeName]",
"types": [ "login" ]
}]
}
where [themeName] is my theme name.
Keycloak is running with 3 instances, its resource spec includes:
extensions:
- [URL-to-jar]
Deployment was successful according to the logs of each pod - each log contains a message containing
Deployed "[jar-name].jar" (runtime-name : "[jar-name].jar")
However, in the admin console, I cannot select the theme from the extension for the login-theme. Creating a new realm via crd with a preconfigured login-theme via spec-entry
loginTheme: [themeName]
also does not work - in the admin-console, the selected entry for the login-theme is empty.
I may be missing something basic, but it seems like this ought to work according to this answer if I am not mistaken.
As is so often the case, an uncaught typo was the source of the error.
The directory-structure must not be
META-INF/keycloak-themes.json
themes/[theme-name]/[theme-role]/theme.properties
[...]
But instead
META-INF/keycloak-themes.json
theme/[theme-name]/[theme-role]/theme.properties
[...]
Given a correct structure, keycloak-operator can successfully deploy and load custom-themes as jar-extensions.
We are using mta to structure our application and deploying it using the SAP Cloud SDK Pipeline and Transport Management landscape.
In the mta.yaml, we are referencing the org (organization) parameter value using the placeholder notation ${org}.
The issue is that the org name contains spaces between the characters (viz. Sample Org Name) and that is causing error during the application deployment to Cloud Foundry.
We do not want to rename the org name.
Is there any other placeholder notation that removes the spaces between the characters?
We have observed that ${default-host} removes the spaces from the organization name but its scope is limited to only modules and not resources.
We need the substitution variable in the resources scope.
Appreciate if someone can help us here to resolve the issue.
Please find snippet of the mta.yaml and the error message.
resources:
- name: uaa_test_app
parameters:
path: ./xs-security.json
service-plan: application
service: xsuaa
config:
xsappname: 'test-app-${org}-${space}'
type: org.cloudfoundry.managed-service
Error Message:
Service operation failed: Controller operation failed: 502 Updating service "uaa_test_app" failed: Bad Gateway: Service broker error: Service broker xsuaa failed with: org.springframework.cloud.servicebroker.exception.ServiceBrokerException: Error updating application null (Error parsing xs-security.json data: Inconsistent xs-security.json: Invalid xsappname "Test-App-Sample Org Name-test": May only include characters 'a'-'z', 'A'-'Z', '0'-'9', '_', '-', '', and '/'.)
I'm trying to use the Cache task inside the AzureDevops release pipeline but I'm getting the following error: "Unable to find pipeline caching scopes."
The full log is:
2019-12-04T02:52:29.7320092Z ##[section]Starting: Cache Maven local repo
2019-12-04T02:52:29.7323359Z ==============================================================================
2019-12-04T02:52:29.7324096Z Task : Cache
2019-12-04T02:52:29.7324205Z Description : Cache files between runs
2019-12-04T02:52:29.7324288Z Version : 2.0.0
2019-12-04T02:52:29.7324386Z Author : Microsoft Corporation
2019-12-04T02:52:29.7324476Z Help : https://aka.ms/pipeline-caching-docs
2019-12-04T02:52:29.7324596Z ==============================================================================
2019-12-04T02:52:30.1883169Z Resolving key:
2019-12-04T02:52:30.2148089Z - maven [string]
2019-12-04T02:52:30.2157501Z - "Linux" [string]
2019-12-04T02:52:30.3775363Z - **/pom.xml [file pattern; matches: 1]
2019-12-04T02:52:30.3776471Z - /home/vsts/work/r1/a/_GHQ_ABI_Salesforce_OneApp/AutomationTestsPrj/... --> 14132C0725424D58200724C825592A7666659E231F87619340789DA110453697
2019-12-04T02:52:30.3840201Z Resolved to: maven|"Linux"|eoJcSbaPKzXrAAmPNfX9NFnxcoydrNu9yrhaLikHrF8=
2019-12-04T02:52:30.3891079Z Resolving restore key:
2019-12-04T02:52:30.3891618Z - maven [string]
2019-12-04T02:52:30.3892061Z - "Linux" [string]
2019-12-04T02:52:30.3892473Z Resolved to: maven|"Linux"|**
2019-12-04T02:52:30.3892596Z Resolving restore key:
2019-12-04T02:52:30.3892858Z - maven [string]
2019-12-04T02:52:30.3892979Z Resolved to: maven|**
2019-12-04T02:52:31.7549257Z Information, ApplicationInsightsTelemetrySender will correlate events with X-TFS-Session 6a8d4cb9-7af8-4915-8dcc-46b9ded1e10b
2019-12-04T02:52:32.0675474Z Information, Getting a pipeline cache artifact with one of the following fingerprints:
2019-12-04T02:52:32.0676246Z Information, Fingerprint: `maven|"Linux"|eoJcSbaPKzXrAAmPNfX9NFnxcoydrNu9yrhaLikHrF8=`
2019-12-04T02:52:32.0676528Z Information, Fingerprint: `maven|"Linux"|**`
2019-12-04T02:52:32.0676755Z Information, Fingerprint: `maven|**`
2019-12-04T02:52:34.2400712Z Information, ApplicationInsightsTelemetrySender correlated 2 events with X-TFS-Session 6a8d4cb9-7af8-4915-8dcc-46b9ded1e10b
2019-12-04T02:52:34.2485748Z ##[error]Unable to find pipeline caching scopes.
2019-12-04T02:52:34.2528126Z ##[section]Finishing: Cache Maven local repo
And my configuration is:
Oh, the variables are created and this task is before the task that I execute the Maven:
Cache Task order
So, anybody got an idea of where I'm missing?
I know applying this Cache step before your Maven execution is very convenient for pipeline, such as reducing the whole execution time. Just, unfortunately, until now, we haven't provide this Cache task supported in Release pipeline:
Pipeline caching
Work around:
As I mentioned above, Cache task is very convenient because it can help reduce the time waste in file caching.
So, suggest you apply your release pipeline into Multi-stage YAML format. Though it is running a release phase, but its compile logic is following build.
Check this doc to convert your pipeline: YAML schema.
I have an ARM template that deploys Key Vault and populates it with secrets. It does creates secrets, based on how many arrays are in the parameter secretsObject. For example if I have:
"secretsObject": {
"type": "secureObject",
"defaultValue": {
"secrets": [
{
"secretName": "exampleSecret1",
"secretValue": "secretVaule1"
},
{
"secretName": "exampleSecret2",
"secretValue": "secretValue2"
}
]
}
}
The template will create 2 Secrets. So this is the line that I put into .parameters.json to deploy the template from Visual Studio:
"secrets": [
{
"secretName": "exampleSecret1",
"secretValue": "secretVaule1"
},
{
"secretName": "exampleSecret2",
"secretValue": "secretValue2"
}
]
The problem is I can't figure out how to past such line into VSTS as a variable (to overwrite parameter). This is the ARM template I'm using
There were errors in your deployment. Error code: InvalidDeploymentParameterKey.
One of the deployment parameters has an empty key. Please see https://aka.ms/arm-deploy/#parameter-file for details.
Processed: ##vso[task.issue type=error;]One of the deployment parameters has an empty key. Please see https://aka.ms/arm-deploy/#parameter-file for details.
task result: Failed
Task failed while creating or updating the template deployment.
There is the issue in Azure Resource Group deployment task and I submit a feedback here: VSTS build/release task: Override template parameters of Azure Resource Group Deployment.
The workaround is that you can update the parameter file during the build/release (e.g. parameter.json) and specify this parameter file in Azure Resource Group deployment task.
There are many ways to update file, such as Replace Tokens.
Update:
Feedback in Gitgub: https://github.com/Microsoft/vsts-tasks/issues/6108
We are using some community custom resource types (https://github.com/ljfranklin/terraform-resource and https://github.com/cloudfoundry/bosh-deployment-resource). After upgrading to concourse 3.3.0, we've begun consistently seeing the following error on a few of our jobs at the same step: json: unsupported type: map[interface {}]interface {}.
This is fairly hard to debug as there is no other log output other than that. We are unsure what is incompatible between those resources and Concourse.
Notes about our pipeline:
We originally had substituted all of our usages of {{}} to (()), but reverting that did not lead to the error going away.
We upgraded concourse from v3.0.1.
The failing step can be found here: https://github.com/cloudfoundry/capi-ci/blob/6a73764d09f544820ce39f16dca166d6d6861996/ci/pipeline.yml#L731-L739
We are using a resource called elsa-aws-storage-terraform, found here: https://github.com/cloudfoundry/capi-ci/blob/6a73764d09f544820ce39f16dca166d6d6861996/ci/pipeline.yml#L731-L739
That resource is of a custom resource-type terraform found here: https://github.com/cloudfoundry/capi-ci/blob/6a73764d09f544820ce39f16dca166d6d6861996/ci/pipeline.yml#L45-L48
A similar failing step can be found here: https://github.com/cloudfoundry/capi-ci/blob/6a73764d09f544820ce39f16dca166d6d6861996/ci/pipeline.yml#L871-L886
This is related to issue of not being able to define nested maps in resource configuration https://github.com/concourse/concourse/issues/1345