Update Stack w. "Replacement: Conditional"; will it replace? - aws-cloudformation

Just updated some of our CF-templates and was going to update the stack it refers to. I've added some default values and added a CloudWatch alarm and I am also going to downgrade the instance from m4.xlarge to m4.large.
I've already downgraded the instance in the EC2-GUI and it went fine. I then reverted it to its default state as per the original template i.e. m4.xlarge.
However when I modify the default value in the template for the instancetype it does not reflect when I upload the modified template to CloudFormation.
Meaning the default value is still m4.xlarge and I have to use the drop-down menu to select m4.large as specified in my template.
If I don't change the instancetype I get:
"Replacement: False" but if I update the instancetype I get "Replacement: Conditional".
If I read more under "Changeset Details" and then "Details" I see:
[
{
"resourceChange": {
"logicalResourceId": "CPUAlarm",
"action": "Add",
"physicalResourceId": null,
"resourceType": "AWS::CloudWatch::Alarm",
"replacement": null,
"details": [],
"scope": []
},
"type": "Resource"
},
{
"resourceChange": {
"logicalResourceId": "myInstanceName",
"action": "Modify",
"physicalResourceId": "<masked>",
"resourceType": "AWS::EC2::Instance",
"replacement": "Conditional",
"details": [
{
"target": {
"name": null,
"requiresRecreation": "Never",
"attribute": "Tags"
},
"causingEntity": null,
"evaluation": "Dynamic",
"changeSource": "DirectModification"
},
{
"target": {
"name": null,
"requiresRecreation": "Never",
"attribute": "Tags"
},
"causingEntity": "Project",
"evaluation": "Static",
"changeSource": "ParameterReference"
},
{
"target": {
"name": null,
"requiresRecreation": "Never",
"attribute": "Tags"
},
"causingEntity": null,
"evaluation": "Static",
"changeSource": null
},
{
"target": {
"name": "InstanceType",
"requiresRecreation": "Conditionally",
"attribute": "Properties"
},
"causingEntity": "InstanceType",
"evaluation": "Static",
"changeSource": "ParameterReference"
},
{
"target": {
"name": "InstanceType",
"requiresRecreation": "Conditionally",
"attribute": "Properties"
},
"causingEntity": null,
"evaluation": "Dynamic",
"changeSource": "DirectModification"
}
],
"scope": [
"Properties",
"Tags"
]
},
"type": "Resource"
},
{
So what I can see is that:
"name": "InstanceType","requiresRecreation": "Conditionally", is the only value that has a more restrictive value and therefore the entire stack gets "Replacement: Conditional".
As per AWS:
"In some cases, AWS CloudFormation can determine a value only after you execute a change set. AWS CloudFormation labels those changes as Dynamic evaluations. For example, if you reference an updated resource that is conditionally replaced, AWS CloudFormation can't determine whether the reference to the updated resource will change."
Source:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-changesets-samples.html#using-cfn-updating-stacks-changesets-samples-directly-editing-a-template
AFAIK "Replacement: Conditional" 'might' replace the resource i.e. creating a new physicalResourceID which in turn forces me to change associated SGs etc. but it might also not do it, correct?
Grateful for any assistance!

It's generally not recommended to modify resources created via CloudFormation, outside of CloudFormation. For your last question, Replacement: Conditional may or may not necessitate replacement of a resource based on what exactly you're trying to do. It's always helpful to look at the AWS CloudFormation docs whenever you have doubts though e.g. in your specific scenario of editing instance type of an EC2 instance, here's the what docs state:
Update requires: Some interruptions for Amazon EBS-backed instances
Update requires: Replacement for instance store-backed instances
Ref: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html#cfn-ec2-instance-instancetype

Related

Azure DevOps KeyVault Linked Variable group "Value cannot be null. Parameter name: variableGroupParameters" - how do i fix this?

I have a need to automate the creation of KeyVault linked variable groups in ADO as part of a pipeline task. I can actually create the var groups after a bit of experimenting. However, using the az devops invoke method one is unable to specify the Azure Subscription and this has to be done manually, after the event - however when I do this in the web interface and attempt to save I get the error:
Value cannot be null. Parameter name: variableGroupParameters
This means any subsequent editing of the created KeyVault linked var group is pointless as it is unable to save it.
The JSON I am submitting is as follows:
{
"authorized": true,
"description": "$description",
"name": "$name",
"type": "AzureKeyVault",
"variableGroupProjectReferences": [{
"projectReference": {
"id": "$adoProjectID",
"name": "$ProjectName"
},
"name": "$name",
"description": "$description"
}],
"providerData": {
"serviceEndpointId": "$AdoSvcConnId",
"vault": "$KeyVaultNM"
},
"variables": {
"SOMETHINGSECRET": {
"isSecret": true,
"value": null,
"enabled": true,
"contentType": ""
},
"variables": {
"ANOTHERSECRET": {
"isSecret": true,
"value": null,
"enabled": true,
"contentType": ""
}
}
}
}
Where the $tokenized values are replaced during the powershell/az cli task
The command (which works but results in broken var group that can't be edited or saved) is as follows:
az devops invoke --http-method post --area distributedtask --resource variablegroups --in-file "vgroup_azure_rm.json" --encoding utf-8 --route-parameters project=$ProjectName --api-version 5.0-preview
Anyone have an insight in how to fix this please?
I was experiencing the same problem, and the message had nothing to do with the real issue.
I had the following variables:
"variables": {
"SECRET_1": {
"contentType": "",
"isSecret": true,
"value": "",
"expires": null,
"enabled": true
},
"SECRET_2": {
"contentType": "",
"isSecret": true,
"value": "",
"expires": null,
"enabled": true
},
"SECRET_3": {
"contentType": "",
"isSecret": true,
"value": "",
"expires": null,
"enabled": true
}
}
And was receiving the same error as you. I removed the expires property and the issue was solved! So my thinking is that the validation checks for any null value and returns the error you are receiving.
Try changing the property value to an empty string ("") instead of null and see whether that solves the problem.

Sonarqube REST API : What is the structure of "metrics" in "GET api/measures/component" WS

In the response example from the web api documentation, I can see metric "ncloc" should be like this in the JSON web response :
"measures": [
{
"metric": "ncloc",
"value": "114",
"periods": [
{
"index": 1,
"value": "3"
}
]
},
But it's not, there is no "periods" for this metric in my response :
"measures":[
{
"metric": "ncloc",
"value": "2943"
},
There are "periods" for some other metrics though, and in this case, there is no metric value, only a value for each period (and there are never multiple periods, only one corresponding to the "new code" period).
So here are my questions about this :
How can I know which structure to expect for a metric ? what would be the metric where "periods" would have a list of periods and not just one corresponding to "new code" ?
I don't think there are multiple periods. API documentation for my version (8.9) only states period, which probably stands for Leak Period as the New Code used to be called. I assume that general value is for overall value and period value is for the new code. Some measures may not make sense or be counted for overall or new code, so I would not make assumptions on whether there will be a metric value for the period or general.
Edit:
The following is the 8.9 documentation:
GET api/measures/component
SINCE 5.4
Return component with specified measures.
Requires the following permission: 'Browse' on the project of specified component.
Parameters
Parameter
Required?
Since
Description
Example value
additionalFields
optional
Comma-separated list of additional fields that can be returned in the response.
Possible values: metrics, period, periods Example value: period,metrics
branch
optional
6.6
Branch key. Not available in the community edition.
Example value: feature/my_branch
component
required
Component key
Example value: my_project
metricKeys
required
Comma-separated list of metric keys
Example value: ncloc,complexity,violations
pullRequest
optional
7.1
Pull request id. Not available in the community edition.
Example value: 5461
Response Example
{
"component": {
"key": "MY_PROJECT:ElementImpl.java",
"name": "ElementImpl.java",
"qualifier": "FIL",
"language": "java",
"path": "src/main/java/com/sonarsource/markdown/impl/ElementImpl.java",
"measures": [
{
"metric": "complexity",
"value": "12",
"period": {
"value": "2",
"bestValue": false
}
},
{
"metric": "new_violations",
"period": {
"value": "25",
"bestValue": false
}
},
{
"metric": "ncloc",
"value": "114",
"period": {
"value": "3",
"bestValue": false
}
}
]
},
"metrics": [
{
"key": "complexity",
"name": "Complexity",
"description": "Cyclomatic complexity",
"domain": "Complexity",
"type": "INT",
"higherValuesAreBetter": false,
"qualitative": false,
"hidden": false,
"custom": false
},
{
"key": "ncloc",
"name": "Lines of code",
"description": "Non Commenting Lines of Code",
"domain": "Size",
"type": "INT",
"higherValuesAreBetter": false,
"qualitative": false,
"hidden": false,
"custom": false
},
{
"key": "new_violations",
"name": "New issues",
"description": "New Issues",
"domain": "Issues",
"type": "INT",
"higherValuesAreBetter": false,
"qualitative": true,
"hidden": false,
"custom": false
}
],
"period": {
"mode": "previous_version",
"date": "2016-01-11T10:49:50+0100",
"parameter": "1.0-SNAPSHOT"
}
}
Changelog
Version
Change
8.8
deprecated response field 'id' has been removed.
8.8
deprecated response field 'refId' has been removed.
8.1
the response field periods under measures field is deprecated. Use period instead.
8.1
the response field periods is deprecated. Use period instead.
7.6
The use of module keys in parameter 'component' is deprecated
6.6
the response field 'id' is deprecated. Use 'key' instead.
6.6
the response field 'refId' is deprecated. Use 'refKey' instead.

How to create idempotent, re-deployable ARM templates that utilize Key Vault? Circular dependencies present issues

I'm trying to incorporate some additional security features on my resources such as encryption using customer managed keys. For the Service Bus, this requires the Service Bus to have a managed identity created and granted access to the Key Vault. However, the managed identity is not known until after the Service Bus exists.
In my ARM template, I need to initialize the Service Bus without encryption in order to get a managed identity, grant that identity access to key vault, then update the Service Bus with encryption. However, this deployment process is not repeatable. On subsequent re-deployments, it will fail because encryption cannot be removed from a Service Bus once it is granted. Even if it did work, I would be performing unnecessary steps to remove encryption and add it back on every deployment. It seems that an IAC template that describes the expected final state of the resource cannot be used to both maintain and bootstrap a new environment.
The same problem exists for API Management, I want to add custom domains but they require Key Vault access. It means I can't re-deploy my ARM template without removing the custom domains when the init step gets repeated (or keep a separate set of templates for 'initialization' vs the 'real deployment'.
Is there a better solution for this? I looked into user assigned identities which seems like it could solve the problem, but they aren't supported in ARM templates. I checked if there is a way to make the 'init' step conditional by checking if the resource already exists, and this is also not supported via ARM template.
If you want to maintain a single ARM template, you can use nested deployments to define a resource then reference it again to update it.
In the following example, I create the Service Bus with a system-assigned managed identity, a Key Vault, and an RSA key. In a nested deployment within the same ARM template that depends on that key being generated, I then update the Service Bus to enable encryption. An advantage of being all in the same template is that resourceId and reference can be used with abbreviated syntax (i.e. just the resource name).
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"name": {
"type": "string",
"defaultValue": "[resourceGroup().name]"
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]"
},
"tenantId": {
"type": "string",
"defaultValue": "[subscription().tenantId]"
}
},
"variables": {
"kv_name": "[concat(parameters('name'), 'kv')]",
"kv_version": "2019-09-01",
"sb_name": "[concat(parameters('name'), 'sb')]",
"sb_version": "2018-01-01-preview",
"sb_keyname": "sbkey"
},
"resources": [
{
"type": "Microsoft.ServiceBus/namespaces",
"apiVersion": "[variables('sb_version')]",
"name": "[variables('sb_name')]",
"location": "[parameters('location')]",
"sku": {
"name": "Premium",
"tier": "Premium",
"capacity": 1
},
"identity": {
"type": "SystemAssigned"
},
"properties": {
"zoneRedundant": false
}
},
{
"type": "Microsoft.KeyVault/vaults",
"apiVersion": "[variables('kv_version')]",
"name": "[variables('kv_name')]",
"location": "[parameters('location')]",
"dependsOn": [
"[variables('sb_name')]"
],
"properties": {
"sku": {
"family": "A",
"name": "Standard"
},
"tenantId": "[parameters('tenantId')]",
"accessPolicies": [
{
"tenantId": "[reference(variables('sb_name'), variables('sb_version'), 'Full').identity.tenantId]",
"objectId": "[reference(variables('sb_name'), variables('sb_version'), 'Full').identity.principalId]",
"permissions": {
"keys": [
"get",
"wrapKey",
"unwrapKey"
]
}
}
],
// Both must be enabled to encrypt Service Bus at rest.
"enableSoftDelete": true,
"enablePurgeProtection": true
}
},
{
"type": "Microsoft.KeyVault/vaults/keys",
"apiVersion": "[variables('kv_version')]",
"name": "[concat(variables('kv_name'), '/', variables('sb_keyname'))]",
"location": "[parameters('location')]",
"dependsOn": [
"[variables('kv_name')]"
],
"properties": {
"kty": "RSA",
"keySize": 2048,
"keyOps": [
"wrapKey",
"unwrapKey"
],
"attributes": {
"enabled": true
}
}
},
{
"type": "Microsoft.Resources/deployments",
"apiVersion": "2020-10-01",
"name": "sb_deployment",
"dependsOn": [
"[resourceId('Microsoft.KeyVault/vaults/keys', variables('kv_name'), variables('sb_keyname'))]"
],
"properties": {
"mode": "Incremental",
"template": {
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"resources": [
{
"type": "Microsoft.ServiceBus/namespaces",
"apiVersion": "[variables('sb_version')]",
"name": "[variables('sb_name')]",
"location": "[parameters('location')]",
"sku": {
"name": "Premium",
"tier": "Premium",
"capacity": 1
},
"identity": {
"type": "SystemAssigned"
},
"properties": {
"zoneRedundant": false,
"encryption": {
"keySource": "Microsoft.KeyVault",
"keyVaultProperties": [
{
// Ideally should specify a specific version, but no ARM template function to get this currently.
"keyVaultUri": "[reference(variables('kv_name')).vaultUri]",
"keyName": "[variables('sb_keyname')]"
}
]
}
}
}
]
}
}
}
]
}
To deploy this using the Azure CLI:
az group create -g rg-example -l westus2
az deployment group create -g rg-example -f template.json --parameters name=example

How do I use an extended definition and not allow additional properties in a way that is compatible with multiple validators (JSON schema draft 7)?

I am creating a strict validator for a complex JSON file and want to re-use various definitions in order to keep the schema manageable and easier to update.
According to the documentation it is necessary to use allOf to extend a definition to add more properties. This is exactly what I've done, but I find that without use of additionalProperties set to false validation doesn't prevent arbitrary other properties being added.
The following massively cut-down schema demonstrates what I'm doing:
{
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://example.com/schema/2021/02/example.json",
"description": "This schema demonstrates how VSCode's JSON schema mechanism fails with allOf used to extend a definition",
"definitions": {
"valueProvider": {
"type": "object",
"properties": {
"example": {
"type": "string"
},
"alternative": {
"type": "string"
}
},
"oneOf": [
{
"required": [
"example"
]
},
{
"required": [
"alternative"
]
}
]
},
"selector": {
"type": "object",
"allOf": [
{
"$ref": "#/definitions/valueProvider"
},
{
"required": [
"operator",
"value"
],
"properties": {
"operator": {
"type": "string",
"enum": [
"IsNull",
"Equals",
"NotEquals",
"Greater",
"GreaterOrEquals",
"Less",
"LessOrEquals"
]
},
"value": {
"type": "string"
}
}
}
],
"additionalProperties": false
}
},
"properties": {
"show": {
"properties": {
"name": {
"type": "string"
},
"selector": {
"description": "This property does not function correctly in VSCode",
"allOf": [
{
"$ref": "#/definitions/selector"
},
{
"additionalProperties": false
}
]
}
},
"additionalProperties": false
}
}
}
This works a treat in IntelliJ IDEA's JSON editor (2020.3.2 ultimate edition) when editing JSON against this schema (using a schema mapping). For example, the file ex-fail.json's content of:
{
"show": {
"name": "a",
"selector": {
"example": "a",
"operator": "IsNull",
"value": "false",
"d": "a"
}
}
}
Is correctly validated, simply highlighting "d" as not allowed, thus:
However, when I use the very same schema and JSON file with VSCode (1.53.2) with vanilla configuration (except for a schema mapping) VSCode erroneously marks "example", "operator", "value" and "d" as not allowed. It looks like this in the VSCode editor:
If I remove the additionalProperties definition from the show.selector property, both IDEA and VSCode indicate that all is well, including allowing the "d" property - in doing this I can simplify that property definition to:
"selector": {
"description": "This property does not function correctly in VSCode",
"$ref": "#/definitions/selector"
}
What can I do to the schema to support both IDEA and VSCode whilst disallowing additional properties where they should not appear?
PS: The schema mapping in VSCode is simply along the lines of:
{
"json.schemas": [
{
"fileMatch": [
"*/config/ex-*.json"
],
"url": "file:///C:/my/path/to/example-schema.json"
}
]
}
You cannot do what you ask with JSON Schema draft-07 or prior.
The reason is, when $ref is used in a schema object, all other properties MUST be ignored.
An object schema with a "$ref" property MUST be interpreted as a
"$ref" reference. The value of the "$ref" property MUST be a URI
Reference. Resolved against the current URI base, it identifies the
URI of a schema to use. All other properties in a "$ref" object MUST
be ignored.
https://datatracker.ietf.org/doc/html/draft-handrews-json-schema-01#section-8.3
We changed this to not be the case for draft 2019-09.
It sounds like VSCode is merging the properties in applicators upwards to the nearest schema object (which is wrong), and IntelliJ IDEA is doing something similar but in a different way (which is also wrong).
The correct validation result for your schema and instance is VALID. See the live demo here: https://jsonschema.dev/s/C6ent
additionalProperties relies on the values of properties and patternProperties within the SAME schema object. It cannot "see through" applicators such as $ref and allOf.
For draft 2019-09, we added unevaluatedProperties, which CAN "see through" applicator keywords (although it's a little more complex than that).
Update:
After reviewing your update, sadly the same is still true.
One approach makes it sort of possible but involves some duplication, and only works when you control the schemas you are referencing.
You would need to redefine your selector property like this...
"selector": {
"description": "This property did not function correctly in VSCode",
"allOf": [
{
"$ref": "#/definitions/selector"
},
{
"properties": {
"operator": true,
"value": true,
"example": true,
"alternative": true
},
"additionalProperties": false
}
]
}
The values of a property object are schema values, and booleans are valid schemas. You don't need (or want to) deal with their validation here, only say these are the allowed ones, followed by no additionalProperties.
You'll also need to remove the additionalProperties: false from your definition of selector, as that is preventing ALL properties (which I now guess is why you saw that issue in one of the editors).
It involves some duplication, but is the only way I'm aware of that you can do this for draft-07 or previous. As I said, not a problem for draft 2019-09 or above due to new kewords.
additionalProperties is problematic because it depends on the properties and patternProperties. The result is that "additionalProperties": false effectively blocks schema composition. #Relequestual showed one alternative approach, here is another approach that is a little less verbose, but still requires duplication of property names.
draft-06 and up
{
"allOf": [{ "$ref": "#/definitions/base" }],
"properties": {
"bar": { "type": "number" }
},
"propertyNames": { "enum": ["foo", "bar"] },
"definitions": {
"base": {
"properties": {
"foo": { "type": "string" }
}
}
}
}

Azure DevOps API - how to discover link between field and picklist

I'm trying to replicate an Azure DevOps process from one organization to another via the AZDO REST Api. I'm working on replicating the layout and am stuck because I can't discover the relationship between a custom field and a picklist when querying the source AZDO instance.
In my scenario I have a test work item type which I've called Issue. On the Issue interface I've created a custom field which is a picklist. While I can retrieve a list of lists via the Rest API and examine the field as well, I can't figure out how the two are related.
Here is a partial payload from the field:
{
"count": 39,
"value": [
...
{
"referenceName": "Custom.IssueSource",
"name": "Issue Source",
"type": "string",
"description": "Who is this attributed to",
"required": true,
"url": "https://dev.azure.com/MYORG/_apis/work/processes/f390103e-7097-4f19-b5b5-f9dbcf92bb6f/behaviors",
"customization": "custom"
},
... ]
}
and here is a partial payload from the lists get query which I used trial and error to determine was the picklist I've assigned:
{
"count": 10,
"value": [
...
{
"id": "2998d4e4-2bec-4935-98a1-b67a0b0b6d5d",
"name": "picklist_e854661e-8620-4ad9-be28-b974c5cb3a5d",
"type": "String",
"isSuggested": false,
"url": "https://dev.azure.com/MYORG/_apis/work/processes/lists/2998d4e4-2bec-4935-98a1-b67a0b0b6d5d"
},
...
]
}
Here is a partial layout response for the WIT:
{
"pages": [
{
"id": "d0171d51-ff84-4038-afc1-8800ab613160.System.WorkItemType.Details",
"inherited": true,
"label": "Details",
"pageType": "custom",
"visible": true,
"isContribution": false,
"sections": [
{
"id": "Section1",
"groups": [
...
{
"id": "bf03e049-5062-4d82-b91d-4396541fbed2",
"label": "Custom",
"isContribution": false,
"visible": true,
"controls": [
{
"id": "Custom.IssueSource",
"label": "Issue Source",
"controlType": "FieldControl",
"readOnly": false,
"visible": true,
"isContribution": false
}
]
}
]
},
... ]
}
Using fiddler against the AZDO web interface, the only time I see a reference to the picklist is from another non-AZDO API to https://dev.azure.com/MYORG/_apis/Contribution/dataProviders/query
Is there a way to discover the link via the AZDO Rest API? I saw this question which was similar but was about creating the link
Figured it out. Turns out you need to query from a different scope - work item tracking rather than work item tracking process:
https://dev.azure.com/MYORG/_apis/wit/fields/Custom.IssueSource?api-version=5.0-preview.2
returns
{
"name": "Issue Source",
"referenceName": "Custom.IssueSource",
"description": "Who is this attributed to",
"type": "string",
"usage": "workItem",
"readOnly": false,
"canSortBy": true,
"isQueryable": true,
...
"isIdentity": false,
--> "isPicklist": true,
"isPicklistSuggested": false,
--> "picklistId": "2998d4e4-2bec-4935-98a1-b67a0b0b6d5d",
"url": "https://dev.azure.com/MYORG/_apis/wit/fields/Custom.IssueSource"
}