Azure DevOps pipeline error: Tenant ID, application ID, principal ID, and scope are not allowed to be updated - azure-devops

I try to create SQL Server with ARM on Azure DevOps.
Pipeline successfully create SQL Server resource to Azure Portal, but I'm getting strange errors in Azure DevOps. Why this occurs and how to fix?
ERROR:
There were errors in your deployment. Error code: DeploymentFailed.
##[error]RoleAssignmentUpdateNotPermitted: Tenant ID, application ID, principal ID, and scope are not
allowed to be updated.
##[error]Check out the troubleshooting guide to see if your issue is addressed:
https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment?
view=azure-devops#troubleshooting
##[error]Task failed while creating or updating the template deployment.
YML:
task: AzureResourceManagerTemplateDeployment#3
inputs:
deploymentScope: 'Resource Group'
azureResourceManagerConnection: 'TestRG-Conn'
subscriptionId: '1111753a-501e-4e46-9aff-6120ed561111'
action: 'Create Or Update Resource Group'
resourceGroupName: 'TestRG'
location: 'North Europe'
templateLocation: 'Linked artifact'
csmFile: '$(System.DefaultWorkingDirectory)/CreateSQLServer/azuredeploy.json'
csmParametersFile:
'$(System.DefaultWorkingDirectory)/CreateSQLServer/azuredeploy.parameters.json'
deploymentMode: 'Incremental'
VARIABLE IN TEMPLATE:
"variables": {
"StorageBlobContributor": "[subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '111111111111111111111-')]"
},
RESOURCE IN TEMPLATE:
"resources": [
{
"condition": "[parameters('enableADS')]",
"type":
"Microsoft.Storage/storageAccounts/providers/roleAssignments",
"apiVersion": "2018-09-01-preview",
"name": "[concat(variables('storageName'),
'/Microsoft.Authorization/', variables('uniqueRoleGuid') )]",
"dependsOn": [
"[resourceId('Microsoft.Sql/servers',
parameters('serverName'))]",
"[resourceId('Microsoft.Storage/storageAccounts',
variables('storageName'))]"
],
"properties": {
"roleDefinitionId": "[variables('StorageBlobContributor')]",
"principalId": "[reference(resourceId('Microsoft.Sql/servers',
parameters('serverName')), '2018-06-01-preview',
'Full').identity.principalId]",
"scope": "[resourceId('Microsoft.Storage/storageAccounts',
variables('storageName'))]",
"principalType": "ServicePrincipal"
}
}

Chances are you have deployed and deleted the resources, however, the role assignment is still there and that is what it is clashing with (what 4c7... is saying). So, go check the permissions on the storage account - if you use managed identities, that identity will be deleted but the role assignment will persists and show the user as 'unknown' which will also cause the above error when trying to deploy again - had the same issue but with a managed identity I was using for an aks cluster. Frustrating.
When you deleted a managed identity it does not delete associated roles created for it, I wish it cleaned up properly.

In my case, it was the name of the RoleAssignment. It was unique on the Resource Group level but not on the subscription level. Not sure what is the scope for the uniqueness of the name.

Bouncing off #Richard answer, I didn't have the permission to delete the "ghost" managed identities so I deployed the same role assignment under a different guid by adding an additional string to the guid() function. String functions for ARM templates docs
To do this, I changed my roleNameGuid's value from
"[guid(resourceGroup().id)]" to
"[guid(resourceGroup().id, parameters('guid_seed'))]", where parameters('guid_seed') is an arbitrary string that is passed from DevOps.

Related

Create policy with CloudFormation

I am able to create a policy this way with the AWS CLI:
aws iam create-policy --policy-name "alpha-policy" --policy-document file:///tmp/policy.json
The content of the policy.json is following:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Statement",
"Effect": "Allow",
"Action": [ "s3:ListBucket"],
"Resource": "*"
}
]
}
I convert it into the following CloudFormation file:
Resources:
SimplePolicy:
Type: AWS::IAM::Policy
Properties:
PolicyName: OfficialSimplePolicy
PolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Resource: "*"
Action:
- cloudformation:Describe*
And the command to create the policy with CloudFormation is:
aws cloudformation create-stack --stack-name bucket-policy --template-body file://BucketPolicy.yaml --capabilities CAPABILITY_IAM
When I run this command, I only get a stack ID back. However, no policy is created. What am I missing?
I would have expected the policy to be available in the aws console or via following command:
aws iam list-policies
It's nowhere to be found.
I checked the event list with:
aws cloudformation describe-stack-events --stack-name bucket-policy
What this reveals is that: "At least one of [Groups,Roles,Users] must be non-empty."
And so my question is why can I create a policy without user, group or role when using the cli directly and I am not able to do the same when using cloudformation.
Following article explains my problem: https://cloudkatha.com/iam-policy-at-least-one-of-groupsrolesusers-must-be-non-empty/
Basically for standalone policies I should use ManagedPolicy. Also PolicyName is not a support field. These two changes solved my problem.

Deploying azure storage fileServices/shares - error: The value for one of the HTTP headers is not in the correct format

As part of a durable function app deployment, I am deploying azure storage.
On deploying the fileServices/shares, I am getting the following error:
error": {
"code": "InvalidHeaderValue",
"message": "The value for one of the HTTP headers is not in the correct format.\nRequestId:6c0b3fb0-701a-0058-0509-a8af5d000000\nTime:2022-08-04T13:49:24.6378224Z"
}
I would appreciate any advice as this is eating up a lot of time and I am no closer to resolving it.
Section of arm template for the share deployment is below:
{
"type": "Microsoft.Storage/storageAccounts/fileServices/shares",
"apiVersion": "2021-09-01",
"name": "[concat(parameters('storageAccount1_name'), '/default/FuncAppName')]",
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts/fileServices', parameters('storageAccount1_name'), 'default')]",
"[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccount1_name'))]"
],
"properties": {
"accessTier": "TransactionOptimized",
"shareQuota": 5120,
"enabledProtocols": "SMB"
}
}
Answer to this: removing the property "accessTier": "TransactionOptimized" resolves the issue. The default value for this is TransactionOptimized.
Although the template exported from azure portal includes this property, deployment fails if this parameter is present.

'Required property 'type' not found in JSON. Path 'properties.template.parameters.factoryName', line 1, position 208.'

I'm attempting to publish our development ADF objects from our development subscription to our test subscription.
To accomplish this, I used the Microsoft guide to set up a release pipeline. My final result looks like this:
However, when attempting to deploy to test, it fails:
2022-05-04T12:22:05.6519328Z ##[section]Starting: ARM Template deployment: Resource Group scope 2022-05-04T12:22:05.6923741Z
============================================================================== 2022-05-04T12:22:05.6925480Z Task : ARM template deployment 2022-05-04T12:22:05.6925967Z Description : Deploy an Azure Resource Manager (ARM) template to all the deployment scopes 2022-05-04T12:22:05.6926257Z Version : 3.200.0 2022-05-04T12:22:05.6926589Z Author : Microsoft Corporation 2022-05-04T12:22:05.6927048Z Help : https://learn.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment 2022-05-04T12:22:05.6927663Z
============================================================================== 2022-05-04T12:22:06.5933183Z ARM Service Connection deployment scope - Subscription 2022-05-04T12:22:06.6193437Z Checking if the following resource group exists: tst-<Removed>. 2022-05-04T12:22:06.8901626Z Resource group exists: true. 2022-05-04T12:22:06.8904567Z Creating deployment parameters. 2022-05-04T12:22:06.9013317Z The detected encoding for file 'D:\a\r1\a\Dev\dev-adf-<Removed>\ARMTemplateParametersForFactory.json' is 'utf-8' 2022-05-04T12:22:06.9064165Z (node:4736) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead. 2022-05-04T12:22:07.0403949Z Starting template validation. 2022-05-04T12:22:07.0478298Z Deployment name is ARMTemplateParametersForFactory-20220504-122207-af92 2022-05-04T12:22:07.3412253Z There were errors in your deployment. Error code: InvalidRequestContent. 2022-05-04T12:22:07.3458400Z
##[error]The request content was invalid and could not be deserialized: 'Required property 'type' not found in JSON. Path 'properties.template.parameters.factoryName', line 1, position 208.'. 2022-05-04T12:22:07.3475499Z ##[warning]Validation errors were found in the Azure Resource Manager template. This can potentially cause template deployment to fail. Task failed while creating or updating the template deployment.. Please follow https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/template-syntax 2022-05-04T12:22:07.3477016Z Starting Deployment. 2022-05-04T12:22:07.3477522Z Deployment name is ARMTemplateParametersForFactory-20220504-122207-af92 2022-05-04T12:22:07.4293852Z There were errors in your deployment. Error code: InvalidRequestContent. 2022-05-04T12:22:07.4295964Z
##[error]The request content was invalid and could not be deserialized: 'Required property 'type' not found in JSON. Path 'properties.template.parameters.factoryName', line 1, position 208.'. 2022-05-04T12:22:07.4298521Z ##[error]Check out the troubleshooting guide to see if your issue is addressed: https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/azure-resource-group-deployment?view=azure-devops#troubleshooting 2022-05-04T12:22:07.4301572Z ##[error]Task failed while creating or updating the template deployment. 2022-05-04T12:22:07.4465058Z
##[section]Finishing: ARM Template deployment: Resource Group scope
The contents of the ARMTemplateParametersForFactory.json:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"factoryName": {
"value": "dev-adf-<removed>"
},
"<removed>_EDW_connectionString": {
"value": ""
},
"dev_blob_<removed>_connectionString": {
"value": ""
}
}
}
As you can see, there is no line 208 in that file. The logs are not helpful at all, and to the best of my knowledge, I have not made any changes to the parameter file.
I have tried reading related questions, but they all seem to deal with different issues or are not related to Data Factory projects. Checking other files within the adf_publish branch, I did see some of them had a "type":"string" property. I attempted to manually add that to the ArmTemplateParametersForDataFactory file as well, but the error remains.
I'm franmkly not sure how to proceed. Any help would be appreciated. Thank you.

AWS ECS Blue/Green CodePipeline: Exception while trying to read the image artifact

I wanted to create a CodePipeline which builds a container image from CodeCommit source and afterwards deploys the new image in Blue/Green fashion to my ECS service (EC2 launchtype).
The source stage is CodeCommit, which already includes appspec.json
as well as taskdef.json
The build stage is building the new
container & pushing it to ECR successfully, the file
imagedefinition.json is the BuildArtifact created at this step,
containing the container and the recently created image with its
tag corresponding to the CodeCommit commit-id.
The deploy stage
is made of action "Amazon ECS (Blue/Green)" using the
SourceArtifact and BuildArtifact as InputArtifacts, to take the
appspec and taskdef from the SourceArtifact and the image
description from the BuildArtifact, to finally deploy the new
container in Blue/Green manner.
The problem is with the image definition from the BuildArtifact. The pipeline fails in the Deploy phase with error:
""
Invalid action configuration
Exception while trying to read the image artifact file from the artifact: BuildArtifact.
""
How to properly configure the "Amazon ECS (Blue/Green)" deploy phase, so that it can use the recently created image and deploy it....by replacing placeholder IMAGE_NAME inside taskdef.json ?
Any hint highly appreciated :D
answering my own question here, hopefully it helps others who facing the same situation.
the file imagedefinitions.json is inappropriate for deploy action "Amazon ECS Blue/Green". For that you have to create file imageDetail.json within the build step and provide it as artifact to the deploy step. How ? This is how the bottom of my buildspec.yaml looks like:
- printf '{"ImageURI":"%s"}' $REPOSITORY_URI:$IMAGE_TAG > imageDetail.json
artifacts:
files:
- 'image*.json'
- 'appspec.yaml'
- 'taskdef.json'
secondary-artifacts:
DefinitionArtifact:
files:
- appspec.yaml
- taskdef.json
ImageArtifact:
files:
- imageDetail.json
In the Deploy phase of CodePipeline, use DefinitionArtifact and ImageArtifact as Input Artifacts and configure them in the corresponding section "Amazon ECS task definition" and "AWS CodeDeploy AppSpec file".
Ensure that your appspec.yaml contains placeholder for the task definition. Here is my appspec.yaml:
version: 0.0
Resources:
- TargetService:
Type: AWS::ECS::Service
Properties:
TaskDefinition: <TASK_DEFINITION>
LoadBalancerInfo:
ContainerName: "my-test-container"
ContainerPort: 8000
Also ensure that your taskdef.json contains placeholder for the final image, like
...
"image": <IMAGE1_NAME>,
...
use that placeholder in the codepipeline config of your blue/green deploy phase in the section "Dynamically update task definition image - optional" by choosing the input artifact as "ImageArtifact" and the placeholder <IMAGE1_NAME>
Amazon ECS Blue/Green (or CodeDeployToECS) CodePipeline action requires the TaskDefinitionTemplateArtifact parameter (see [1]).
In addition to the above file note an imageDetail.json is required for ECS Blue/Green deployments (not 'imagedefinition.json'). The file structure and details are available here [2]. Add this file to the root of your deployment artifact/version control. If you do not want to add this file manually you can use the ECR source action to the CodePipeline and configure this with the Image you are using in the ECS service/taskdef.json. This is all discussed at [2] for clarity.
To see how this is all brought together you can also follow the step by step instructions for ECS Blue/Green deployments here [3].
References:
[1] https://docs.aws.amazon.com/codepipeline/latest/userguide/reference-pipeline-structure.html#action-requirements : CodePipeline Pipeline Structure Reference - Action Structure Requirements in CodePipeline
[2] https://docs.aws.amazon.com/codepipeline/latest/userguide/file-reference.html#file-reference-ecs-bluegreen : Image Definitions File Reference - imageDetail.json File for Amazon ECS Blue/Green Deployment Actions
[3] https://docs.aws.amazon.com/codepipeline/latest/userguide/tutorials-ecs-ecr-codedeploy.html : Tutorial: Create a Pipeline with an Amazon ECR Source and ECS-to-CodeDeploy Deployment
I ran into the same problem.
tl:dr
I was not passing the correct input artefact with the imageDetail.json to the pipeline CodeDeployToECS action.
Summary:
Instead of checking in a version of the task definition with the '<IMAGE1_NAME>' placeholder, I'm dynamically generating the task definition input to CodeDeploy inside the pipeline.
The task definition early in the project is quite volatile, with new variables etc being passed to the container. It's generated and registered within the pipeline (Cloudformation) and then read out via a Codebuild project, substituting the image placeholder with '<IMAGE1_NAME>' and passed to the next stage in the pipeline via a pipeline artefact.
Fixing it:
I have a CodeBuild project within the pipeline that produces the imageDetail.json:
{"ImageURI":"########.dkr.ecr.eu-west-1.amazonaws.com/##/#####:2739511dd87d4e4e1f65ed69c9e779b63fb72e36-master-fbe73fdc-6213-4bd6-a784-dcc3d2ae7845"}
It's pipeline output is named 'BuildDockerOutput'
I have another Codebuild project that produces:
taskdef.json
{
"containerDefinitions": [
{
"name": "ronantest1",
"image": "<IMAGE1_NAME>",
]
}
appspec.json
{
"version": 0.0,
"Resources": [
{
"TargetService": {
"Type": "AWS::ECS::Service",
"Properties": {
"TaskDefinition": "<TASK_DEFINITION>",
"LoadBalancerInfo": {
"ContainerName": "ronantest1",
"ContainerPort": "8080"
}
}
}
}
],
"Hooks": [
{
"AfterAllowTestTraffic": "arn:aws:lambda:eu-west-1:######:function:code-deploy-after-allow-test-traffic"
}
]
}
It's pipeline output is named 'PrepareCodeDeployOutputTesting'
My final CodeDeploy action is like the following:
- Name: BlueGreenDeploy
InputArtifacts:
- Name: BuildDockerOutput
- Name: PrepareCodeDeployOutputTesting
Region: !Ref DeployRegion1
ActionTypeId:
Category: Deploy
Owner: AWS
Version: '1'
Provider: CodeDeployToECS
RoleArn: !Sub arn:aws:iam::${TestingAccountId}:role/######/CrossAccountsDeploymentRole
Configuration:
AppSpecTemplateArtifact: PrepareCodeDeployOutputTesting
AppSpecTemplatePath: appspec.json
ApplicationName: !Ref ApplicationName
DeploymentGroupName: !Ref ApplicationName
TaskDefinitionTemplateArtifact: PrepareCodeDeployOutputTesting
TaskDefinitionTemplatePath: taskdef.json
Image1ArtifactName: BuildDockerOutput
Image1ContainerName: "IMAGE1_NAME"
RunOrder: 4
Note the different aspects of the CodeDeployToECS needed artefacts from different InputArtifacts, specifically 'Image1ArtifactName'
Thanks, to all, this gives me some light into solving the issue.
I would like to add that when you use aws cli, cloudformation, or Terraform to configure codepipeline, some parameters and options are not available with the console and setting some variables in these tools like the empty string "" will cause an exception error.
Always check for codepipeline settings in the console when you deploy using these tools.
so the error occur when you defined Image Artifact but not define the placeholder
imageDetail.json can be passed into codedeploy using the following methods:
git source ( codecommit or github ) the file that exist in your app codebase
ECR source - the file will be autogenerated by ECR, but will use SHA256 instead of the image tag
CodeBuild source - you update the file using codebuild buildspec.yml and pass it down to codedeploy stage.

Passing secureObject array as VSTS variable

I have an ARM template that deploys Key Vault and populates it with secrets. It does creates secrets, based on how many arrays are in the parameter secretsObject. For example if I have:
"secretsObject": {
"type": "secureObject",
"defaultValue": {
"secrets": [
{
"secretName": "exampleSecret1",
"secretValue": "secretVaule1"
},
{
"secretName": "exampleSecret2",
"secretValue": "secretValue2"
}
]
}
}
The template will create 2 Secrets. So this is the line that I put into .parameters.json to deploy the template from Visual Studio:
"secrets": [
{
"secretName": "exampleSecret1",
"secretValue": "secretVaule1"
},
{
"secretName": "exampleSecret2",
"secretValue": "secretValue2"
}
]
The problem is I can't figure out how to past such line into VSTS as a variable (to overwrite parameter). This is the ARM template I'm using
There were errors in your deployment. Error code: InvalidDeploymentParameterKey.
One of the deployment parameters has an empty key. Please see https://aka.ms/arm-deploy/#parameter-file for details.
Processed: ##vso[task.issue type=error;]One of the deployment parameters has an empty key. Please see https://aka.ms/arm-deploy/#parameter-file for details.
task result: Failed
Task failed while creating or updating the template deployment.
There is the issue in Azure Resource Group deployment task and I submit a feedback here: VSTS build/release task: Override template parameters of Azure Resource Group Deployment.
The workaround is that you can update the parameter file during the build/release (e.g. parameter.json) and specify this parameter file in Azure Resource Group deployment task.
There are many ways to update file, such as Replace Tokens.
Update:
Feedback in Gitgub: https://github.com/Microsoft/vsts-tasks/issues/6108