Azure Automation Registration Endpoint is corrupted when used to pull DSC configuration - powershell

For some reason, I keep getting these weird issues.....
In this case, I have a Key and Endpoint URL for the Automation Account stored as Secrets in a KeyVault (I don't know of a away to extract it natively from Automation Account using ARM).
I can extract these values perfectly and they they are published to the Template that runs a PowerShell extension to pull a DSC Configuration.
For example as seen as an Input deploying the Template:
"RegistrationUrl":"https://ase-agentservice-prod-1.azure-automation.net/accounts/e0799801-a8da-8934-b0f3-9a43191dd7e6"
However, I receive the following error (note the Url in the Error with 3 forward slashes)
"code": "VMExtensionProvisioningError",
"message": "VM has reported a failure when processing extension 'dscLcm'.
Error message: "DSC Configuration 'ConfigureLCMforAAPull' completed with error(s). Following are the first few: The attempt to 'get an action' for AgentId 11A5A267-6D00-11E7-B07F-000D3AE0FB1B from server URL https://ase-agentservice-prod-1.azure-automation.net///accounts/e0799801-a8da-8934-b0f3-9a43191dd7e6/Nodes(AgentId='11A5A267-6D00-11E7-B07F-000D3AE0FB1B')/GetDscAction failed with server error 'ResourceNotFound(404)'.
For further details see the server error message below or the DSC debug event log with ID 4339.
ServerErrorMessage:- 'No NodeConfiguration was found for the agent.'\"."
The Endpoint Url is passed as a Secure String. I tried passing it a normal string - Same problem.
The Key and Endpoint are feed into the Template as Parameters:
"dscKeySecret": {
"type": "securestring",
"metadata": {
"description": "Key for PowerShell DSC Configuration."
}
},
"dscUrlSecret": {
"type": "securestring",
"metadata": {
"description": "Url for PowerShell DSC Configuration."
}
},
These values are used to create a parameter to be passed to the next template that runs the VM Extension.
"extn-settings": {
"value": {
"configuration": {
"url": "[concat(variables('urls').dscScripts, '/', 'lcm-aa-pull', '/', 'lcm-aa-pull', '.zip')]",
"script": "[concat('lcm-aa-pull', '.ps1')]",
"function": "ConfigureLCMforAAPull"
},
"configurationArguments": {
"registrationKey": {
"username": "dsckeySecret",
"password": "[parameters('dscKeySecret')]"
},
"registrationUrl": "[parameters('dscUrlSecret')]",
"configurationMode": "ApplyAndMonitor",
"configurationModeFrequencyMins": 15,
"domain": "[variables('names').domain]",
"name": "dscLcm",
"nodeConfigurationName": "[variables('names').config.ad]",
"rebootNodeIfNeeded": true,
"refreshFrequencyMins": 30
},
"protectedSettings": null,
}
}
The next template receives the Parameters and used in the Properties of the VM's Resources section:
"properties": {
"publisher": "Microsoft.Powershell",
"type": "DSC",
"typeHandlerVersion": "2.22",
"autoUpgradeMinorVersion": true,
"settings": {
"configuration": "[parameters('extn-settings').configuration]",
"configurationArguments": "[parameters('extn-settings').configurationArguments]"
},
"protectedSettings": "[parameters('extn-settings').protectedSettings]"
}
So why is the Url being corrupted with the the first '/' being changed to '///'?

I don't why the Endpoint Url has 3 x '/', but that wasn't the issue.... I wish I found the issue before I posted this question...
I found the Node Configuration Name was wrong with a spelling mistake (hang head in shame)
Thanks anyway!

Related

ADF: linkedService template function not defined

I am currently trying to add some parameterised linked services. I have two services currently: a key vault, and a data lake. The configuration are:
// Key vault
{
"name": "Logical Key Vault",
"properties": {
"parameters": {
"environment": {
"type": "String"
}
},
"annotations": [],
"type": "AzureKeyVault",
"typeProperties": {
"baseUrl": "https://kv-#{linkedService().environment}.vault.azure.net"
}
}
}
// Data lake
{
"name": "Logical Data Lake",
"properties": {
"type": "AzureBlobFS",
"parameters": {
"environment": {
"type": "String"
}
},
"annotations": [],
"typeProperties": {
"url": "https://sa#{replace(linkedService().environment, '-', '')}.dfs.core.windows.net",
"accountKey": {
"type": "AzureKeyVaultSecret",
"secretName": "storageAccountKey",
"store": {
"referenceName": "Logical Key Vault",
"type": "LinkedServiceReference",
"parameters": {
"environment": {
"value": "#linkedService().environment",
"type": "Expression"
}
}
}
}
}
}
}
Both linked services are parameterised by an environment parameter, and I have confirmed that the Key Vault works fine and is able to correctly retrieve secrets. The problem happens when I attempt to retrieve the storage key from the key vault. I get the following error:
Error code
FailToResolveParametersInExploratoryController
Details
The parameters and expression cannot be resolved for schema operations.
Error Message: {
"message": "ErrorCode=InvalidTemplate, ErrorMessage=The template function 'linkedService' is not defined or not valid."
}
My attempts at debugging this has identified the use of #linkedService on line 38 to be the issue, which is when the Data Lake is trying to pass its own environment parameter to the Key Vault so that it may obtain the storage key. If I remove this use of #linkedService.environment and replace it with a hard coded value, the linked service successfully connects to the data lake.
The expression is trivially simple, and the web interface itself offers the option:
As a result, I am unsure why the use of #linkedService fails here. The web interface and ability to use expressions would suggest it should work, but then #linkedService is undefined for some reason.
While debugging this, I did find that using the expression
#string(linkedService.environment)
Does indeed work, but this seems rather odd as the environment is itself a string and thus its conversion into a string should be a no-op. I have also looked into removing the # entirely and trying
linkedService.environment
and while this does correctly resolve to the environment, it still results in an error as it resulting parameter contains the surrounding quotation and thus the linked service fails to connect to the key vault https://kv-'foobar'.vault.azure.net as it is clearly invalid (assuming my environment was foobar).

Arm template deployment fail with 409 error for one specific storage account

I use arm template to deploy a storage account. However, I got an error saying: StorageAccountAlreadyExists: The storage account named xxx already exists.
My release pipeline is set to incremental, so shouldn't really show this error.
I changed storage account name to a new one, not only it worked the first time, but I can keep on deploying the same pipeline and no error ever thrown out.
Looks like it is something specific to this account, however, I can't see anything special. The arm template we use is also quite normal (something we got from official examples before).
{
"$schema": "http://schema.management.azure.com/schemas/2019-06-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"StorageDescriptor": {
"type": "string",
"defaultValue": "StorageAccount",
"metadata": {}
},
"StorageAccountName": {
"type": "string",
"defaultValue": "[toLower(concat(parameters('StorageDescriptor'), resourceGroup().name))]",
"metadata": { "Description": "Override name for the storage account" }
},
"StorageType": {
"type": "string",
"defaultValue": "Standard_LRS",
"allowedValues": [
"Standard_LRS",
"Standard_ZRS",
"Standard_GRS",
"Standard_RAGRS",
"Premium_LRS"
]
},
"Environment": {
"type": "string",
"defaultValue": "PreProd",
"metadata": { "description": "PreProd or Prod" }
}
},
"variables": {
},
"resources": [
{
"name": "[parameters('StorageAccountName')]",
"type": "Microsoft.Storage/storageAccounts",
"location": "[resourceGroup().location]",
"apiVersion": "2019-06-01",
"dependsOn": [],
"tags": {
"displayName": "Web Job Storage Account"
},
"properties": {
"accountType": "[parameters('StorageType')]"
}
}
],
"outputs": {
}
}
Even though your release pipeline is set to incremental, the storage account name must be unique every time you deploy. Refer to: here.
Arm template deployment fail with 409 error for one specific storage account
You need to check if the storage account attributes have been changed through the Azure/PowerShell portal by somebody else, and are different than the ones specified on the ARM template.
To resolve this issue, please try to export the template and update it in the Azure devops repo:
Then, we could update this new exported template file as you want and deploy with it.
As test, I could keep on deploying the same pipeline and no error ever thrown out.

Add/remove pipeline checks using REST API

I have a requirement to dynamically add/remove or enable disable approval and checks at azure DevOps pipeline environment.
Is there a rest api for this?
Is there a rest api for this?
Yes, it has. BUT, as what the #Krysztof said, we haven't provide such api documents to public as of today. This is because what you are looking for is one feature (configure Check and approval from environments ) that only support for YAML pipeline, and until now, we are developing but does not publish the corresponding rest api (for YAML) docs.
But, as work around, you can catch these apis from F12. Just do action from UI, then capture and analyze corresponds api records to found out what you are expecting.
Here I make the summary to you.
The api that used to add/delete approval and check to environment is:
https://dev.azure.com/{org name}/{project name}/_apis/pipelines/checks/configurations
The corresponding api method for add or delete is Post and DELETE.
Firstly, share you the request body samples which for approval and check added.
1) Add approval to this environment:
{
"type": {
"id": "8C6F20A7-A545-4486-9777-F762FAFE0D4D", // The fixed value for Approval
"name": "Approval"
},
"settings": {
"approvers": [
{
"id": "f3c88b9a-b49f-4126-a4fe-3c99ecbf6303" // User Id
}
],
"executionOrder": 1,
"instructions": "",
"blockedApprovers": [],
"minRequiredApprovers": 0,
"requesterCannotBeApprover": false // The pipeline requester allow to approve it.
},
"resource": {
"type": "environment",
"id": "1", // Environment id
"name": "Deployment" //Environment name
},
"timeout": 43200 // Set the available time(30d) of this approval pending. The measure unit is seconds.
}
2) Add task check, Azure function, Invoke rest api task and etc:
{
"type": {
"id": "fe1de3ee-a436-41b4-bb20-f6eb4cb879a7", // Fixed value if you want to add task check
"name": "Task Check" //Fixed value
},
"settings": {
"definitionRef": {
"id": "537fdb7a-a601-4537-aa70-92645a2b5ce4", //task Id
"name": "AzureFunction", //task name
"version": "1.0.10" //task version
},
"displayName": "Invoke Azure Function", //task display name configured
"inputs": {
"method": "POST",
"waitForCompletion": "false",
"function": "csdgsdgsa",
"key": "436467543756" // These are all task inputs
},
"retryInterval": 5, // The re-try time specified.
"linkedVariableGroup": "AzKeyGroup"// The variable group name this task linked with
},
"resource": {
"type": "environment",
"id": "2",
"name": "Development"
},
"timeout": 43200
}
In this request body, you can find the corresponding task id from our public source code. Just check the task.json file of corresponding task.
3) Add template check:
{
"type": {
"id": "4020E66E-B0F3-47E1-BC88-48F3CC59B5F3", // Fixed value for template check added.
"name": "ExtendsCheck" //Fixed value
},
"settings": {
"extendsChecks": [
{
"repositoryType": "git", // github for Github source, bitbucket for Bitbucket source
"repositoryName": "MonnoPro",
"repositoryRef": "refs/heads/master",
"templatePath": "tem.yml"
}
]
},
"resource": {
"type": "environment",
"id": "6",
"name": "development"
}
}
In this body, if the template source is coming from github or bitbucket, the value of repositoryName should like {org name}/{repos name}.
Hope these are helps.
This is an older question but we had a similar need. There does not appear to be a direct API To query this, but this GitHub Project pointed us in the right direction:
# GET ENVIRONMENT CHECKS (stored under .fps.dataProviders.data['ms.vss-pipelinechecks.checks-data-provider'].checkConfigurationDataList)
GET https://dev.azure.com/{{organization}}/{{project}}/_environments/{{environment_id}}/checks?__rt=fps&__ver=2
As mentioned above under .fps.dataProviders.data['ms.vss-pipelinechecks.checks-data-provider'].checkConfigurationDataList the list of who is authorized is provided.
The officially documented APIs can tell you that there are checks in place; for example:
GET https://dev.azure.com/{organization}/{project}/_apis/pipelines/checks/configurations?resourceType=environment&resourceId={id}
Can tell you that you have checks enabled (including an Approval check) but this isn't super useful as it does not give a list of who can Approve.
Note that you can get the list of environments (to get their resource ID) using this documented API:
GET https://dev.azure.com/{organization}/{project}/_apis/distributedtask/environments?api-version=7.1-preview.1
This is not supported at the moment. You can upvote feature request to show your interest here.

Deploying an Azure VM and Users with an ARM template and DSC

I am taking my first look at creating a DSC (Desired State Configuration) to go with an ARM (Azure Resource Manager) template to deploy a Windows Server 2016 and additional local user accounts. So far the ARM template works fine and for the DSC file I am using simple example to test functionality. The deployment works fine until I try to pass a username/password so I can create a local Windows user account. I can't seem to make this function work at all (see the error message below).
My question is, how do I use the ARM template to pass the credentials (password) to the DSC (mof) file so that the user can be created without having to explicitly allow plain text passwords (which is not a good practice)?
This is what I have tried:
DSC file
Configuration xUser_CreateUserConfig {
[CmdletBinding()]
Param (
[Parameter(Mandatory = $true)]
[string]
$nodeName,
[Parameter(Mandatory = $true)]
[System.Management.Automation.PSCredential]
[System.Management.Automation.Credential()]
$Credential
)
Import-DscResource -ModuleName xPSDesiredStateConfiguration
Node $nodeName {
xUser 'CreateUserAccount' {
Ensure = 'Present'
UserName = Split-Path -Path $Credential.UserName -Leaf
Password = $Credential
}
}
}
Azure ARM Template Snippet 1st Method
"resources": [
{
"apiVersion": "2016-03-30",
"type": "extensions",
"name": "Microsoft.Powershell.DSC",
"location": "[parameters('location')]",
"tags": {
"DisplayName": "DSC",
"Dept": "[resourceGroup().tags['Dept']]",
"Created By": "[parameters('createdBy')]"
},
"dependsOn": [
"[resourceId('Microsoft.Compute/virtualMachines', concat(variables('vmNamePrefix'), copyIndex(1)))]"
],
"properties": {
"publisher": "Microsoft.Powershell",
"type": "DSC",
"typeHandlerVersion": "2.19",
"autoUpgradeMinorVersion": true,
"settings": {
"wmfVersion": "latest",
"modulesUrl": "[concat(variables('_artifactslocation'), '/', variables('dscArchiveFolder'), '/', variables('dscArchiveFileName'))]",
"configurationFunction": "xCreateUserDsc.ps1\\xUser_CreateUserConfig",
"properties": {
"nodeName": "[concat(variables('vmNamePrefix'), copyIndex(1))]",
"Credential": {
"UserName": "[parameters('noneAdminUsername')]",
"Password": "PrivateSettingsRef:UserPassword"
}
}
},
"protectedSettings": {
"Items": {
"UserPassword": "[parameters('noneAdminUserPassword')]"
}
}
}
}
]
Error message
The resource operation completed with terminal provisioning state 'Failed'. VM has reported a failure when processing extension 'Microsoft.Powershell.DSC'. Error message: \\"The DSC Extension received an incorrect input: Compilation errors occurred while processing configuration 'xUser_CreateUserConfig'. Please review the errors reported in error stream and modify your configuration code appropriately. System.InvalidOperationException error processing property 'Password' OF TYPE 'xUser': Converting and storing encrypted passwords as plain text is not recommended. For more information on securing credentials in MOF file, please refer to MSDN blog: http://go.microsoft.com/fwlink/?LinkId=393729
This error message does not help
Azure ARM Template snippet 2nd Method
"resources": [
{
"apiVersion": "2018-10-01",
"type": "extensions",
"name": "Microsoft.Powershell.DSC",
"location": "[parameters('location')]",
"tags": {
"DisplayName": "DSC",
"Dept": "[resourceGroup().tags['Dept']]",
"Created By": "[parameters('createdBy')]"
},
"dependsOn": [
"[resourceId('Microsoft.Compute/virtualMachines', concat(variables('vmNamePrefix'), copyIndex(1)))]"
],
"properties": {
"publisher": "Microsoft.Powershell",
"type": "DSC",
"typeHandlerVersion": "2.9",
"autoUpgradeMinorVersion": true,
"settings": {
"wmfVersion": "latest",
"configuration": {
"url": "[concat(variables('_artifactslocation'), '/', variables('dscArchiveFolder'), '/', variables('dscArchiveFileName'))]",
"script": "xCreateUserDsc.ps1",
"function": "xUser_CreateUserConfig"
},
"configurationArguments": {
"nodeName": "[concat(variables('vmNamePrefix'), copyIndex(1))]"
},
"privacy": {
"dataCollection": "Disable"
}
},
"protectedSettings": {
"configurationArguments": {
"Credential": {
"UserName": "[parameters('noneAdminUsername')]",
"Password": "[parameters('noneAdminUserPassword')]"
}
}
}
}
}
]
Error Message
VM has reported a failure when processing extension 'Microsoft.Powershell.DSC'. Error message: "The DSC Extension received an incorrect input: A parameter cannot be found that matches parameter name '$credential.Password'. Another common error is to specify parameters of type PSCredential without an explicit type. Please be sure to use a typed parameter in DSC Configuration, for example: configuration Example param([PSCredential] $UserAccount). Please correct the input and retry executing the extension. More information on troubleshooting is available at https://aka.ms/VMExtensionDSCWindowsTroubleshoot
This does not help!
I have been trying to solve this error for a couple of days. I have Googled for other example but can only find example of people deploying Web Server and Microsoft's documentation is no help because it tells you to use both of the above methods. When method 1 is the old way (according to Microsoft). So, any help will be much appreciated.
this is how I was setting up parameter in the configuration:
# Credentials
[Parameter(Mandatory)]
[System.Management.Automation.PSCredential]$Admincreds,
and then in the template:
"properties": {
"publisher": "Microsoft.Powershell",
"type": "DSC",
"typeHandlerVersion": "2.19",
"autoUpgradeMinorVersion": true,
"settings": {
"configuration": xxx // doesn't matter for this question
"configurationArguments": yyy // doesn't matter for this question
},
"protectedSettings": {
"configurationArguments": {
"adminCreds": {
"userName": "someValue",
"password": "someOtherValue"
}
}
}
}
Links to working stuff:
https://github.com/Cloudneeti/PCI_Reference_Architecture/blob/master/templates/resources/AD/azuredeploy.json#L261
https://github.com/Cloudneeti/PCI_Reference_Architecture/blob/master/artifacts/configurationscripts/ad-domain.ps1#L11
ps. you might also need to do this. Honestly, I dont remember ;)

Creating a domain in Plesk's REST API

So, experimenting with Plesk's REST API (available as of version 17.8) for a project at work, and I'm starting to get a feel for it. I've been trying to experiment with adding a domain, but it's been giving me errors when I have to specify the hosting type.
The request body itself is as follows:
{
"name":"example.com",
"hosting_type":"virtual",
"description":"Description goes here"
}
This gets the following cryptic response:
{
"code": 1014,
"message": "htype\/vrt_hst is specified but there is no hosting\/vrt_hst"
}
Per the documentation provided at /api/v2/swagger.yml, any of the following values should be allowed: virtual, standard_forwarding, frame_forwarding, none
No matter what I put in, however, I get a variant of the response above (htype\/{type} is specified but there is no hosting\/{type}).
At this point I'm kind of stuck; I'm not sure what to check, and any references when I try to look up the error code go to references on Plesk's XML API instead. What's the missing link here needed to get the request to work?
It looks like system user is not specified - hosting_settings. Try to add domain with full json request. Here is example:
{
"name": "example.com",
"description": "My website",
"hosting_type": "virtual",
"hosting_settings": {
"ftp_login": "test_login",
"ftp_password": "test_pwd"
},
"base_domain": {
"id": 7,
"name": "a10-52-41-48.qa.plesk.ru",
"guid": "b623e93d-dc72-4102-b5f0-ded427cf0fb1"
},
"parent_domain": {
"id": 7,
"name": "a10-52-41-48.qa.plesk.ru",
"guid": "b623e93d-dc72-4102-b5f0-ded427cf0fb1"
},
"owner_client": {
"id": 7,
"login": "a10-52-41-48.qa.plesk.ru",
"guid": "b623e93d-dc72-4102-b5f0-ded427cf0fb1",
"external_id": "b623e93d-dc72-4102-b5f0-ded427cf0fb1"
},
"ipv4": [
"212.192.122.46"
],
"ipv6": [
"2002:5bcc:18fd:c:123:123:123:123"
],
"plan": {
"name": "Unlimited"
}
}
Examples for REST API https://app.swaggerhub.com/apis/plesk/api/v2#/Domains/post_domains