Azure REST API does not return encryption settings for Virtual Machine - rest

I have a 16.04-LTS Ubuntu Virtual Machine in my Azure account and I am trying Azure Disk Encryption for this virtual machine making use of this azure cli sample script. On running the encryption script, the azure portal shows its OS disk is encrypted. There is Enabled under Encryption header.
However, the Azure REST API (api link) for getting information about the virtual machine does not return the encryptionSettings under properties.storageProfile.osDisk. I tried both Model View and Model View and Instance View for the api-version 2017-03-30 as well as 2017-12-01. Here is the partial response from the API:
{
"name": "ubuntu",
"properties": {
"osProfile": {},
"networkProfile": {},
"storageProfile": {
"imageReference": {
"sku": "16.04-LTS",
"publisher": "Canonical",
"version": "latest",
"offer": "UbuntuServer"
},
"osDisk": {
"name": "ubuntu-OsDisk",
"diskSizeGB": 30,
"managedDisk": {
"storageAccountType": "Premium_LRS",
"id": "..."
},
"caching": "ReadWrite",
"createOption": "FromImage",
"osType": "Linux"
},
"dataDisks": []
},
"diagnosticsProfile": {},
"vmId": "",
"hardwareProfile": {
"vmSize": "Standard_B1s"
},
"provisioningState": "Succeeded"
},
"location": "eastus",
"type": "Microsoft.Compute/virtualMachines",
"id": ""
}
But for my other encrypted windows virtual machine, I get the correct response which contains encryptionSettings in properties.storageProfile.osDisk:
{
"name": "win1",
"properties": {
"osProfile": {},
"networkProfile": {},
"storageProfile": {
"imageReference": {
"sku": "2016-Datacenter-smalldisk",
"publisher": "MicrosoftWindowsServer",
"version": "latest",
"offer": "WindowsServer"
},
"osDisk": {
"name": "win1_OsDisk_1",
"diskSizeGB": 31,
"managedDisk": {
"storageAccountType": "Premium_LRS",
"id": "..."
},
"encryptionSettings": {
"diskEncryptionKey": {
"secretUrl": "...",
"sourceVault": {
"id": "..."
}
},
"keyEncryptionKey": {
"keyUrl": "...",
"sourceVault": {
"id": "..."
}
},
"enabled": true
},
"caching": "ReadWrite",
"createOption": "FromImage",
"osType": "Windows"
},
"dataDisks": []
},
"diagnosticsProfile": {},
"vmId": "...",
"hardwareProfile": {
"vmSize": "Standard_B1s"
},
"provisioningState": "Succeeded"
},
"location": "eastus",
"type": "Microsoft.Compute/virtualMachines",
"id": "..."
}
Why is the Virtual Machine Get API not returning the encryptionSettings for some VMs? Any help would be greatly appreciated.

I create VM using following command.
az vm create \
--resource-group shuivm \
--name shuivm \
--image Canonical:UbuntuServer:16.04-LTS:latest \
--admin-username azureuser \
--generate-ssh-keys
When I use the following API, I could get encryption setting.
https://management.azure.com/subscriptions/**********/resourceGroups/shuivm/providers/Microsoft.Compute/virtualMachines/shuivm?api-version=2017-03-30"
Note: When OS is encrypted successful, I could use API to get encryption setting.

This is because there are two types of at-rest disk encryption for Azure VMs and they are not reported in the same part of the Azure Management API:
Server-Side Encryption: that you can see in the encryptionSettings section of the VM/compute API when you get a vm details. It will show whether you are encypting with a customer managed key or a platform managed key
ADE: Azure Disk Encryption is actually a VM extension and so you can find it in the VM Extension API instead.
see: https://learn.microsoft.com/en-us/rest/api/compute/virtualmachineextensions/list

Related

How to set/update test run owner in Azure Devops Rest API?

I am creating a test run in Azure Devops with Rest API. A sample POST request I make is like:
{
"name": "Test Run Name",
"automated": true,
"plan": {
"id": 11111111,
"name": null,
"url": null,
"state": null,
"iteration": null
},
"pointIds": [
222222222222
],
"build": {
"id": "2222233455",
"buildNumber": "buildNumberjlkajdlajsldj",
"uri": "vstfs:///Build/Build/2222222222",
"sourceBranch": "refs/pull/22222/merge",
"definition": {
"id": "2222"
}
},
"buildConfiguration": {
"id": 3333333,
"number": "buildNumberjlkajdlajsldj",
"uri": "vstfs:///Build/Build/222222222222"
},
"owner": {
"id": "44444444-2222-bbbb-aaaa-1111111111",
"descriptor": "aad.AAAAAAAAAAAAAAAAAAAAAAAAAA"
}
}
But here owner information is ignored and owner is assigned as authorized user of the requester. The requested account have necessary permissions.
Do I make something wrong or owner information cannot be assigned to another user?

How to create idempotent, re-deployable ARM templates that utilize Key Vault? Circular dependencies present issues

I'm trying to incorporate some additional security features on my resources such as encryption using customer managed keys. For the Service Bus, this requires the Service Bus to have a managed identity created and granted access to the Key Vault. However, the managed identity is not known until after the Service Bus exists.
In my ARM template, I need to initialize the Service Bus without encryption in order to get a managed identity, grant that identity access to key vault, then update the Service Bus with encryption. However, this deployment process is not repeatable. On subsequent re-deployments, it will fail because encryption cannot be removed from a Service Bus once it is granted. Even if it did work, I would be performing unnecessary steps to remove encryption and add it back on every deployment. It seems that an IAC template that describes the expected final state of the resource cannot be used to both maintain and bootstrap a new environment.
The same problem exists for API Management, I want to add custom domains but they require Key Vault access. It means I can't re-deploy my ARM template without removing the custom domains when the init step gets repeated (or keep a separate set of templates for 'initialization' vs the 'real deployment'.
Is there a better solution for this? I looked into user assigned identities which seems like it could solve the problem, but they aren't supported in ARM templates. I checked if there is a way to make the 'init' step conditional by checking if the resource already exists, and this is also not supported via ARM template.
If you want to maintain a single ARM template, you can use nested deployments to define a resource then reference it again to update it.
In the following example, I create the Service Bus with a system-assigned managed identity, a Key Vault, and an RSA key. In a nested deployment within the same ARM template that depends on that key being generated, I then update the Service Bus to enable encryption. An advantage of being all in the same template is that resourceId and reference can be used with abbreviated syntax (i.e. just the resource name).
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"name": {
"type": "string",
"defaultValue": "[resourceGroup().name]"
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]"
},
"tenantId": {
"type": "string",
"defaultValue": "[subscription().tenantId]"
}
},
"variables": {
"kv_name": "[concat(parameters('name'), 'kv')]",
"kv_version": "2019-09-01",
"sb_name": "[concat(parameters('name'), 'sb')]",
"sb_version": "2018-01-01-preview",
"sb_keyname": "sbkey"
},
"resources": [
{
"type": "Microsoft.ServiceBus/namespaces",
"apiVersion": "[variables('sb_version')]",
"name": "[variables('sb_name')]",
"location": "[parameters('location')]",
"sku": {
"name": "Premium",
"tier": "Premium",
"capacity": 1
},
"identity": {
"type": "SystemAssigned"
},
"properties": {
"zoneRedundant": false
}
},
{
"type": "Microsoft.KeyVault/vaults",
"apiVersion": "[variables('kv_version')]",
"name": "[variables('kv_name')]",
"location": "[parameters('location')]",
"dependsOn": [
"[variables('sb_name')]"
],
"properties": {
"sku": {
"family": "A",
"name": "Standard"
},
"tenantId": "[parameters('tenantId')]",
"accessPolicies": [
{
"tenantId": "[reference(variables('sb_name'), variables('sb_version'), 'Full').identity.tenantId]",
"objectId": "[reference(variables('sb_name'), variables('sb_version'), 'Full').identity.principalId]",
"permissions": {
"keys": [
"get",
"wrapKey",
"unwrapKey"
]
}
}
],
// Both must be enabled to encrypt Service Bus at rest.
"enableSoftDelete": true,
"enablePurgeProtection": true
}
},
{
"type": "Microsoft.KeyVault/vaults/keys",
"apiVersion": "[variables('kv_version')]",
"name": "[concat(variables('kv_name'), '/', variables('sb_keyname'))]",
"location": "[parameters('location')]",
"dependsOn": [
"[variables('kv_name')]"
],
"properties": {
"kty": "RSA",
"keySize": 2048,
"keyOps": [
"wrapKey",
"unwrapKey"
],
"attributes": {
"enabled": true
}
}
},
{
"type": "Microsoft.Resources/deployments",
"apiVersion": "2020-10-01",
"name": "sb_deployment",
"dependsOn": [
"[resourceId('Microsoft.KeyVault/vaults/keys', variables('kv_name'), variables('sb_keyname'))]"
],
"properties": {
"mode": "Incremental",
"template": {
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"resources": [
{
"type": "Microsoft.ServiceBus/namespaces",
"apiVersion": "[variables('sb_version')]",
"name": "[variables('sb_name')]",
"location": "[parameters('location')]",
"sku": {
"name": "Premium",
"tier": "Premium",
"capacity": 1
},
"identity": {
"type": "SystemAssigned"
},
"properties": {
"zoneRedundant": false,
"encryption": {
"keySource": "Microsoft.KeyVault",
"keyVaultProperties": [
{
// Ideally should specify a specific version, but no ARM template function to get this currently.
"keyVaultUri": "[reference(variables('kv_name')).vaultUri]",
"keyName": "[variables('sb_keyname')]"
}
]
}
}
}
]
}
}
}
]
}
To deploy this using the Azure CLI:
az group create -g rg-example -l westus2
az deployment group create -g rg-example -f template.json --parameters name=example

Azure Blob Storage Deployment: Stored Access Policy gets deleted

Context:
I deploy a storage account as well as one or more containers with the following ARM template with Azure DevOps respectively a Resource Deployment Task:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"storageAccountName": {
"type": "string",
"metadata": {
"description": "The name of the Azure Storage account."
}
},
"containerNames": {
"type": "array",
"metadata": {
"description": "The names of the blob containers."
}
},
"location": {
"type": "string",
"metadata": {
"description": "The location in which the Azure Storage resources should be deployed."
}
}
},
"resources": [
{
"name": "[parameters('storageAccountName')]",
"type": "Microsoft.Storage/storageAccounts",
"apiVersion": "2018-07-01",
"location": "[parameters('location')]",
"kind": "StorageV2",
"sku": {
"name": "Standard_LRS",
"tier": "Standard"
},
"properties": {
"accessTier": "Hot"
}
},
{
"name": "[concat(parameters('storageAccountName'), '/default/', parameters('containerNames')[copyIndex()])]",
"type": "Microsoft.Storage/storageAccounts/blobServices/containers",
"apiVersion": "2018-03-01-preview",
"dependsOn": [
"[parameters('storageAccountName')]"
],
"copy": {
"name": "containercopy",
"count": "[length(parameters('containerNames'))]"
}
}
],
"outputs": {
"storageAccountName": {
"type": "string",
"value": "[parameters('storageAccountName')]"
},
"storageAccountKey": {
"type": "string",
"value": "[listKeys(parameters('storageAccountName'), '2018-02-01').keys[0].value]"
},
"storageContainerNames": {
"type": "array",
"value": "[parameters('containerNames')]"
}
}
}
Input can be e.g.
-storageAccountName 'stor1' -containerNames [ 'con1', 'con2' ] -location 'westeurope'
In an next step I create Stored Access Policies for the containers deployed.
Problem:
The first time I do that everything works fine. But if I execute the pipeline a second time the Stored Access Policies gets deleted by the deployment of the template. The storage account itself with its containers and blobs are not deleted (as it should be). This is unfortunate because I want to keep the Stored Access Policy with its starttime and expirytime as deployed the first time, furthermore I expect that the SAS also become invalid (not tested so far).
Questions:
Why is this happening?
How can I avoid this problem respectively keep the Stored Access Policies?
Thanks
After doing some investigation this seems to be by design. When deploying ARM templates for storage accounts the PUT operation is used, i.e. elements that are not specified within the template are removed. As it is not possible to specify Shared Access Policies for containers within an ARM template for Storage Accounts existing ones get deleted when the template is redeployed...

Creating a Azure VM using Azure Resource Manger API

I am using following azure rest api to create a virtual machine in Azure Resource Manager mode
PUT
https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group-name}/providers/Microsoft.Compute/virtualMachines/{vm-name}?validating={true|false}&api-version={api-version}
The virtual machine gets created and it remains in Creating state as i can see the status or new portal of azure
I am able to RDP into the machine.
But after a login it fails.
What could be the reason?
Any thing i am missing?
Note : I am creating a virtual machine using a image.
Request Json :
{
"properties": {
"hardwareProfile": {
"vmSize": "Standard_A0"
},
"storageProfile": {
"osDisk": {
"osType": "Windows",
"name": "goldentemplate-osDisk",
"createOption": "FromImage",
"image": {
"uri": "https://storagename.blob.core.windows.net/system/Microsoft.Compute/Images/mytemplates/goldentemplate-osDisk.vhd"
},
"vhd": {
"uri": "https://storagename.blob.core.windows.net/vmcontainersagar/sagargoden.vhd"
},
"caching": "None"
}
,
"dataDisks": []
},
"osProfile": {
"computerName": "sagarHostVM",
"adminUsername": "itadmin",
"adminPassword": "Micr0s0ft12!#"
},
"networkProfile": {
"networkInterfaces": [
{
"properties": {},
"id": "/subscriptions/subscritpionid/resourceGroups/harigroup/providers/Microsoft.Network/networkInterfaces/sagarhostnic"
}
]
}
},
"name": "sagarHostVM",
"type": "Microsoft.Compute/virtualMachines",
"location": "WestUs",
"tags": {}
}

Installing Azure powershell in an azure Virtual Machine

I need to write a powershell workflow that creates an Azure Virtual Machine and executes some azure cmdlets in that Azure Virtual Machine. But the newly created VM has no azure powershell module installed in it. My code would be like this
New-AzureQuickVM -Windows -ServiceName $serviceName -Name $vmname -ImageName $VMImage -Password $password -AdminUserName $username -InstanceSize "ExtraSmall" -WaitForBoot
$WinRmUri = Get-AzureWinRMUri -ServiceName $serviceName -Name $vmname
$Cred = new-object -typename System.Management.Automation.PSCredential -argumentlist $username, $password
Invoke-Command -ConnectionUri $WinRmUri -Credential $Cred -ScriptBlock {
Add-AzureAccount ...... ## These cmdlets need Azure Powershell Module
Set-AzureSubscription........
New-AzureStorageAccount......
}
I am not supposed to manually get rdp of that VM and open it to install Azure Powershell Module but to dynamically create a VM using powershell cmdlet and install azure module in that vm using powershell itself.
This can easily be done with an ARM (Azure Resource Manager) template. This is a JSON template which defines objects to be deployed. In your case, you would want to deploy a VM with a custom script extension. Upon provisioning of the VM, the Azure Resource Manager will fetch the supplied files and run your custom powershell. See the example below, and replace the line https://<YOUR-BLOB-HERE>.blob.core.windows.net/resources/CUSTOM-POWERSHELL-SCRIPT.ps1 with your blob and powershell script. To run the script you can use Azure powershell, as described here: https://azure.microsoft.com/en-us/documentation/articles/powershell-azure-resource-manager/
The key cmdlet for your purposes is New-AzureResourceGroup. The invocation will be something like:
Switch-AzureMode -Name AzureResourceManager
New-AzureResourceGroup -Name TestRG1 -Location "West US" -TemplateFile <YOUR-JSON-ARM-TEMPLATE>.json
See a list of ARM templates here for reference: https://github.com/Azure/azure-quickstart-templates . Sample template to modify to run custom code/install Azure powershell.
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"newStorageAccountName": {
"type": "string",
"metadata": {
"description": "Unique DNS Name for the Storage Account where the Virtual Machine's disks will be placed."
}
},
"adminUsername": {
"type": "string",
"metadata": {
"description": "Username for the Virtual Machine."
}
},
"adminPassword": {
"type": "securestring",
"metadata": {
"description": "Password for the Virtual Machine."
}
},
"dnsNameForPublicIP": {
"type": "string",
"metadata": {
"description": "Unique DNS Name for the Public IP used to access the Virtual Machine."
}
},
"windowsOSVersion": {
"type": "string",
"defaultValue": "2012-R2-Datacenter",
"allowedValues": [
"2008-R2-SP1",
"2012-Datacenter",
"2012-R2-Datacenter"
],
"metadata": {
"description": "The Windows version for the VM. This will pick a fully patched image of this given Windows version. Allowed values: 2008-R2-SP1, 2012-Datacenter, 2012-R2-Datacenter."
}
}
},
"variables": {
"location": "West US",
"imagePublisher": "MicrosoftWindowsServer",
"imageOffer": "WindowsServer",
"OSDiskName": "osdiskforwindowssimple",
"nicName": "myVMNic",
"addressPrefix": "10.0.0.0/16",
"subnetName": "Subnet",
"subnetPrefix": "10.0.0.0/24",
"storageAccountType": "Standard_LRS",
"publicIPAddressName": "myPublicIP",
"publicIPAddressType": "Dynamic",
"vmStorageAccountContainerName": "vhds",
"vmName": "MyWindowsVM",
"vmSize": "Standard_A2",
"virtualNetworkName": "MyVNET",
"vnetID": "[resourceId('Microsoft.Network/virtualNetworks',variables('virtualNetworkName'))]",
"subnetRef": "[concat(variables('vnetID'),'/subnets/',variables('subnetName'))]"
},
"resources": [
{
"type": "Microsoft.Storage/storageAccounts",
"name": "[parameters('newStorageAccountName')]",
"apiVersion": "2015-05-01-preview",
"location": "[variables('location')]",
"properties": {
"accountType": "[variables('storageAccountType')]"
}
},
{
"apiVersion": "2015-05-01-preview",
"type": "Microsoft.Network/publicIPAddresses",
"name": "[variables('publicIPAddressName')]",
"location": "[variables('location')]",
"properties": {
"publicIPAllocationMethod": "[variables('publicIPAddressType')]",
"dnsSettings": {
"domainNameLabel": "[parameters('dnsNameForPublicIP')]"
}
}
},
{
"apiVersion": "2015-05-01-preview",
"type": "Microsoft.Network/virtualNetworks",
"name": "[variables('virtualNetworkName')]",
"location": "[variables('location')]",
"properties": {
"addressSpace": {
"addressPrefixes": [
"[variables('addressPrefix')]"
]
},
"subnets": [
{
"name": "[variables('subnetName')]",
"properties": {
"addressPrefix": "[variables('subnetPrefix')]"
}
}
]
}
},
{
"apiVersion": "2015-05-01-preview",
"type": "Microsoft.Network/networkInterfaces",
"name": "[variables('nicName')]",
"location": "[variables('location')]",
"dependsOn": [
"[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'))]",
"[concat('Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'))]"
],
"properties": {
"ipConfigurations": [
{
"name": "ipconfig1",
"properties": {
"privateIPAllocationMethod": "Dynamic",
"publicIPAddress": {
"id": "[resourceId('Microsoft.Network/publicIPAddresses',variables('publicIPAddressName'))]"
},
"subnet": {
"id": "[variables('subnetRef')]"
}
}
}
]
},
},
{
"apiVersion": "2015-05-01-preview",
"type": "Microsoft.Compute/virtualMachines",
"name": "[variables('vmName')]",
"location": "[variables('location')]",
"dependsOn": [
"[concat('Microsoft.Storage/storageAccounts/', parameters('newStorageAccountName'))]",
"[concat('Microsoft.Network/networkInterfaces/', variables('nicName'))]"
],
"properties": {
"hardwareProfile": {
"vmSize": "[variables('vmSize')]"
},
"osProfile": {
"computername": "[variables('vmName')]",
"adminUsername": "[parameters('adminUsername')]",
"adminPassword": "[parameters('adminPassword')]"
},
"storageProfile": {
"imageReference": {
"publisher": "[variables('imagePublisher')]",
"offer": "[variables('imageOffer')]",
"sku" : "[parameters('windowsOSVersion')]",
"version":"latest"
},
"osDisk" : {
"name": "osdisk",
"vhd": {
"uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/',variables('OSDiskName'),'.vhd')]"
},
"caching": "ReadWrite",
"createOption": "FromImage"
}
},
"networkProfile": {
"networkInterfaces": [
{
"id": "[resourceId('Microsoft.Network/networkInterfaces',variables('nicName'))]"
}
]
}
},
"resources": [
{
"name": "CustomScript",
"type": "extensions",
"location": "[variables('location')]",
"apiVersion": "2015-05-01-preview",
"dependsOn": [
"[concat('Microsoft.Compute/virtualMachines/', variables('vmName')]"
],
"properties": {
"publisher": "Microsoft.Compute",
"type": "CustomScriptExtension",
"typeHandlerVersion": "[variables('customScriptExtensionVersion')]",
"settings": {
"fileUris": [
"https://<YOUR-BLOB-HERE>.blob.core.windows.net/resources/CUSTOM-POWERSHELL-SCRIPT.ps1",
"http://go.microsoft.com/?linkid=9811175&clcid=0x409"
],
"commandToExecute": "[concat('powershell.exe -ExecutionPolicy Unrestricted -Command .\\CUSTOM-POWERSHELL-SCRIPT.ps1 -Argument1 argument1')]"
}
}
}
]
}
]
}
If your VM has PowerShell 5.0, then you can use the PowerShell Gallery to install your modules. You will not require any steps mentioned in other answers. All you need to do is write the PowerShell script as you would normally do. Just add the Module from PowerShell gallery using just one cmdlet.
You can either use Install-Module to install a module from the gallery, or you can use Install-Script to install a sample script from the PowerShell public gallery.
You can even put your own modules in the gallery and install from there.
Reference: Get Started with the PowerShell Gallery
You may use Azure Automation service implementing your Powershell code into a runbook.
http://azure.microsoft.com/en-us/documentation/services/automation/
Though not a straight forward approach I implemented this idea that satisfied my need.
Create a new azure VM from Portal and connect it through RDP https://azure.microsoft.com/en-in/documentation/articles/virtual-machines-windows-tutorial/
Now download azure powershell msi in YOUR machine ( In AzureVM downloading is blocked) http://az635501.vo.msecnd.net/azcopy-3-1-0/MicrosoftAzureStorageTools.msi
Manually copy the msi file to the virtual machine and install it in that VM
Now Capture the image of that VM and upload it in Azure My images
https://azure.microsoft.com/en-in/documentation/articles/virtual-machines-capture-image-windows-server/
When I write automation script to create a VM, I used this newly created customVM image, where AzurePowershell is already installed