When I retry unsuccessful operation using Powershell Durable Functions, it throws this exception:
HTTP Trigger script:
using namespace System.Net
param($Request, $TriggerMetadata)
$Request
$FunctionName = $Request.Params.FunctionName
$body = $Request.Body
$InstanceId = Start-NewOrchestration -FunctionName $FunctionName -Input $body
Orchestrator:
param($Context)
$data = $Context.input
Invoke-ActivityFunction -FunctionName 'O365_Holds_Activity' -Input $data
Activity Trigger:
using namespace System.Net
param([System.Collections.Hashtable] $message)
...
And function.json
{
"bindings": [
{
"name": "message",
"type": "activityTrigger",
"direction": "in"
}
],
"retry": {
"strategy": "fixedDelay",
"maxRetryCount": 3,
"delayInterval": "00:00:5"
}
}
First run has no problem with input, but the second one throws such exception. Btw, http request example:
HTTP POST: http://localhost:7071/api/orchestrators/O365_Orchestrator
Body:
{
"holdIds": [
1
],
"action": 0
}
In Durable orchestrators, I would recommend using the Durable-specific retry feature instead of the generic retry policies preview feature, for example:
$retryOptions = New-DurableRetryOptions `
-FirstRetryInterval (New-Timespan -Seconds 5) `
-MaxNumberOfAttempts 3
Invoke-DurableActivity -FunctionName $FunctionName -RetryOptions $retryOptions
The advantage of this approach is that the orchestration state gets persisted between retries, which eliminates the risk of the orchestrator invocation timing out because of retries, and make your orchestration more robust in general.
Related
I have written several Azure Functions over the past year in both powershell and C#. I am currently writing an API that extracts rows from a Storage Account Table and returns that data in a JSON format.
The data pulls fine.
The data converts to JSON just fine.
A JSON formatted response is displayed - which is fine - but the Push-OutputBinding shoves in additional data to my original JSON data - account information, environment information, subscription information, and tenant information.
I've tried a number of different strategies for getting past this. I gave up on using C# to interact with the Tables because the whole Azure.Data.Tables and Cosmos tables packages are a hot mess with breaking changes and package conflicts and .Net 6 requirements for new functions apps. So please don't offer up a C# solution unless you have a working example with specific versions for packages, etc.
Here is the code:
Note that I have verified that $certData and $certJson properly formatted JSON that contain only the data I want to return.
using namespace System.Net
# Input bindings are passed in via param block.
param($Request, $TriggerMetadata)
# Write to the Azure Functions log stream.
Write-Host "PowerShell HTTP trigger function processed a request."
# Interact with query parameters or the body of the request.
$filter = $Request.Query.Filter
if (-not $filter) {
$filter = "ALL"
}
$certData = GetCerts $filter | ConvertTo-Json
#$certJson = $('{ "CertData":"' + $certData + '" }')
$body = "${CertData}"
# Associate values to output bindings by calling 'Push-OutputBinding'.
Push-OutputBinding -Name Response -Value ([HttpResponseContext]#{
StatusCode = [HttpStatusCode]::OK
ContentType = "application/json"
Body = $body
})
When I call the httpTrigger function, the response looks like this:
{ "CertData":"[
{
"Name": "MySubscriptionName blah blah",
"Account": {
"Id": "my user id",
"Type": "User",
....
},
"Environment": {
"Name": "AzureCloud",
"Type": "Built-in",
...
},
"Subscription": {
"Id": "SubscriptionID",
"Name": "SubscriptionName",
....
},
"Tenant": {
"Id": "TenandID",
"TenantId": "TenantId",
"ExtendedProperties": "System.Collections.Generic.Dictionary`2[System.String,System.String]",
...
},
"TokenCache": null,
"VersionProfile": null,
"ExtendedProperties": {}
},
{
"AlertFlag": 1,
"CertID": "abc123",
"CertName": "A cert Name",
"CertType": "an assigned cert type",
"DaysToExpire": 666,
"Domain": "WWW.MYDOMAIN.COM",
"Expiration": "2033-10-04T21:31:03Z",
"PrimaryDomain": "WWW.MYDOMAIN.COM",
"ResourceGroup": "RANDOM-RESOURCES",
"ResourceName": "SOMERESOURCE",
"Status": "OK",
"Subscription": "MYSUBSCRIPTIONNAME",
"Thumbprint": "ABC123ABC123ABC123ABC123ABC123",
"PartitionKey": "PARKEY1",
"RowKey": "ID666",
"TableTimestamp": "2022-02-03T09:00:28.7516797-05:00",
"Etag": "W/\"datetime'2022-02-03T14%3A00%3A28.7516797Z'\""
},
...
Not only does the returned values add data I don't want exposed, it makes parsing the return data that I do want to get when I make API calls problematic.
How do I get rid of the data added by the Push-OutputBinding?
Was able to resolve issue by modifying run.ps1 as follows:
using namespace System.Net
# Input bindings are passed in via param block.
param($Request, $TriggerMetadata)
# Write to the Azure Functions log stream.
Write-Host "PowerShell HTTP trigger function processed a request."
# Interact with query parameters or the body of the request.
$filter = $Request.Query.Filter
if (-not $filter) {
$filter = "ALL"
}
$certData = ( GetCerts $filter | Select-Object -Skip 1 )
#write-information $certData | Format-List
# Associate values to output bindings by calling 'Push-OutputBinding'.
Push-OutputBinding -Name Response -Value ([HttpResponseContext]#{
StatusCode = [HttpStatusCode]::OK
Body = $certData
})
I am trying to update PowerBI datasource using Powershell in Azure devops Pipeline. This works sometimes and sometimes I get an error :
##[error]System.Threading.Tasks.TaskCanceledException: A task was canceled.
The command on which I get this error is :
$urlUpdateDatasource = [string]::Format("groups/{0}/datasets/{1}/Default.UpdateDatasources", $GroupID, $destinationDataset.Id);
$body= "{
""updateDetails"": [
{
""datasourceSelector"": {
""datasourceType"": ""$srcDBType"",
""connectionDetails"": {
""server"": ""$srcServer"",
""database"": ""$srcDB""
}
},
""connectionDetails"": {
""server"": ""$destinationServer"",
""database"": ""$destinationDB""
}
}
]
}"
Write-Host $body
Write-Host $urlUpdateDatasource
try{
$updateDS = Invoke-PowerBIRestMethod -Url $urlUpdateDatasource -Method Post -Body $body
}
catch{
Write-Host 'Failed to UpdateDatasources $_'
}
I have observed that when this step takes for than 1.5 min. I get this error though the update is successful . SO this error is false and shows my pipeline as failed even though it is not. Please suggest how I can handle this.
I have Azure function written in powershell and processing seems to be stopping for no apparent reason with timeout. Message never gets even to start being processed. This does not seem to be code related since code never even have a change to start since the first line of code is Write-Host "PowerShell queue trigger function processed work item: $QueueItem" which does not appear in log
Host.json
{
"version": "2.0",
"functionTimeout": "00:04:00",
"managedDependency": {
"enabled": true
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[1.*, 2.0.0)"
},
"extensions": {
"queues": {
"maxPollingInterval": "00:00:02",
"visibilityTimeout": "00:00:30",
"batchSize": 16,
"maxDequeueCount": 2,
"newBatchThreshold": 8
}
}
}
Function.json
{
"bindings": [
{
"name": "QueueItem",
"type": "queueTrigger",
"direction": "in",
"queueName": "metadataservicequeue",
"connection": "useastbootdiag_STORAGE"
}
]
}
Script
# Input bindings are passed in via param block.
param([string] $QueueItem, $TriggerMetadata)
# Write out the queue message and insertion time to the information log.
Write-Host "PowerShell queue trigger function processed work item: $QueueItem"
Write-Host "Queue item insertion time: $($TriggerMetadata.InsertionTime)"
Write-Host "Starting executing Invoke-AzureRMCommand with parameters $($TriggerMetadata["VMName"]), $($TriggerMetadata["ResourceGroup"])"
$return = Invoke-AzureCommand -vmName $TriggerMetadata["VMName"] -resourceGroup $TriggerMetadata["ResourceGroup"]
Write-Host "Finished executing Invoke-AzureRMCommand with parameters $($TriggerMetadata["VMName"]), $($TriggerMetadata["ResourceGroup"])"
$json = #"
[
{
"Return" : $($return | convertto-json),
"VMName" : "$($TriggerMetadata["VMName"])",
"ResourceGroup" : "$($TriggerMetadata["ResourceGroup"])"
}
]
"#
Write-Host "Outputing following to Log Analytics $json"
Post-LogAnalyticsData -customerId $env:CustomerID -sharedKey $env:SharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($json)) -logType "MetaDataLog"
Powershell Azure Functions by default are not scaling well on consumption plan and poorely scale on dedicated plan due to default settings. If you function is not CPU bound and expected to scale well due to high number of queue items. You'd need to modify FUNCTIONS_WORKER_PROCESS_COUNT (https://learn.microsoft.com/en-us/azure/azure-functions/functions-app-settings) to higher number then default 1 and increase value of PSWorkerInProcConcurrencyUpperBound (https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference-powershell#concurrency) to higher number of default 1
I am using the following azuredeploy.json file for setting up the notification hub on the Azure cloud.
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"Gcm.GoogleApiKey": {
"type": "string",
"metadata": {
"description": "Google Cloud Messaging API Key"
},
"defaultValue": "AIzaSyAyp9MernKgMS3wFNM3yNWByiP-TaGrqEg"
},
"APNS.Certificate": {
"type": "string",
"metadata": {
"description": "A certificate (in base 64 format) provided by Apple on the iOS Provisioning Portal"
},
"defaultValue": ""
},
"APNS.certificateKey": {
"type": "string",
"metadata": {
"description": "The Certificate Key provided by the iOS Provisioning Portal when registering the application"
},
"defaultValue": "ce469bf21dfa7b9d595d4999bfaca8a94ea47e46"
},
"APNS.endpoint": {
"type": "string",
"metadata": {
"description": "The APNS endpoint to which our service connects. This is one of two values: gateway.sandbox.push.apple.com for the sandbox endpoint or gateway.push.apple.com, for the production endpoint. Any other value is invalid."
},
"allowedValues": [
"gateway.sandbox.push.apple.com",
"gateway.push.apple.com"
],
"defaultValue": "gateway.push.apple.com"
}
},
"variables": {
"hubVersion": "[providers('Microsoft.NotificationHubs', 'namespaces').apiVersions[0]]",
"notificationHubNamespace": "[concat('hubv2', uniqueString(resourceGroup().id))]",
"notificationHubName": "notificationhub"
},
"resources": [
{
"name": "[variables('NotificationHubNamespace')]",
"location": "[resourceGroup().location]",
"type": "Microsoft.NotificationHubs/namespaces",
"apiVersion": "[variables('hubVersion')]",
"comments": "Notification hub namespace",
"properties": {
"namespaceType": "NotificationHub"
},
"resources": [
{
"name": "[concat(variables('NotificationHubNamespace'),'/',variables('NotificationHubName'))]",
"location": "[resourceGroup().location]",
"type": "Microsoft.NotificationHubs/namespaces/notificationHubs",
"apiVersion": "[variables('hubVersion')]",
"properties": {
"GcmCredential": {
"properties": {
"googleApiKey": "[parameters('Gcm.GoogleApiKey')]",
"gcmEndpoint": "https://android.googleapis.com/gcm/send"
}
}
},
"dependsOn": [
"[variables('NotificationHubNamespace')]"
]
}
]
}
],
"outputs": {
}
}
Now I tried to set up the apple push notification service also using the following snippet:
"apnsCredential": {
"properties": {
"apnsCertificate": "[parameters('APNS.Certificate')]",
"certificateKey": "[parameters('APNS.certificateKey')]",
"endpoint": " gateway.sandbox.push.apple.com or gateway.push.apple.com",
}
}
With the above changes, I executed the Deploy-AzureResourceGroup.ps1 using powershell command prompt and on executing it I am getting an error with message 'Bad Request'
Can anyone help me to fix this issue.
Add the proper APNS.Certificate and APNS.certificateKey.
It is failing on trying to verify your details, hence the bad request.
You need a base 64 formatted APNS.Certificate.
APNS.Certificate:
This is the Apple Push Notification certificate in base 64 string-format.
You can use PowerShell to convert the certificate like this (then copie the key from the output file ‘MyPushCert.txt’ and use it.):
$fileContentBytes = get-content ‘Apple_Certificate.p12’ -Encoding Byte
[System.Convert]::ToBase64String($fileContentBytes) | Out-File ‘MyPushCert.txt’
APNS.certificateKey:
This is the password you specified when you exported the certificate.(The password you created on Apple at the time of creating the cert.)
It's impossible to know exactly what caused this without knowing more about your environment/setup. According to this post, one possible issue could be how strong your password is:
After a few hours of pulling my hair out and not getting anything
beyond “Bad Request”, I finally thought to use a password stronger
than “pass#word1”. I’ll be darned, it worked. Not only that, but
provisioning with Azure Resource Manager is asynchronous, so your
scripts finish a heck of a lot sooner than they used to because VMs
provision in parallel.
The post recommends going through Troubleshooting common Azure deployment errors with Azure Resource Manager.
I'm not sure you should be dynamically setting the apiVersion for your templates. They vary depending on what you are deploying.
See Best Practices:
Avoid using a parameter or variable for the API version for a resource type. Resource properties and values can vary by version number. IntelliSense in a code editor cannot determine the correct schema when the API version is set to a parameter or variable. Instead, hard-code the API version in the template.
The correct apiVersion for notification hubs appears to be 2015-04-01: https://github.com/Azure/azure-resource-manager-schemas/blob/master/schemas/2015-04-01/Microsoft.NotificationHubs.json
I had to use PowerShell to resolve this problem.
Partially the idea was taken from here: https://learn.microsoft.com/en-us/azure/notification-hubs/notification-hubs-deploy-and-manage-powershell
Below is the script, which tested locally and is working.
Microsoft.Azure.NotificationHubs -Version 1.0.9 used.
We adopted it for VSTS Release as one of the PowerShell tasks/steps after Notification Hub has been created with ARM template.
Login-AzureRmAccount
Select-AzureRmSubscription -SubscriptionName my-subscription-here
Write-Host "Begin process..."
try
{
# Make sure to reference the latest version of Microsoft.Azure.NotificationHubs.dll
Write-Host "Adding the [Microsoft.Azure.NotificationHubs.dll] assembly to the script...";
$scriptPath = Split-Path (Get-Variable MyInvocation -Scope 0).Value.MyCommand.Path;
$packagesFolder = $scriptPath + "\packs";
Write-Host $packagesFolder;
$assembly = Get-ChildItem $packagesFolder -Include "Microsoft.Azure.NotificationHubs.dll" -Recurse;
write-Host $assembly.FullName;
Add-Type -Path $assembly.FullName;
Write-Host "The [Microsoft.Azure.NotificationHubs.dll] assembly has been successfully added to the script.";
# Create requered variables
$HubNamespace = "hub-namespace";
$HubName = "hub-name";
$ResourceGroup = "resource-group";
$GcmApiKey = "api key here";
# Possible values: gateway.push.apple.com, gateway.sandbox.push.apple.com
$ApnsEndpoint = "gateway.push.apple.com";
# A certificate (in base 64 format) provided by Apple on the iOS Provisioning Portal
$ApnsCertificate = "base 64 certificate here";
$ApnsCertificateKey = "certificate key/password here";
$GcmCredential = New-Object -TypeName Microsoft.Azure.NotificationHubs.GcmCredential -ArgumentList $GcmApiKey;
$ApnsCredential = New-Object -TypeName Microsoft.Azure.NotificationHubs.ApnsCredential;
$ApnsCredential.Endpoint = $ApnsEndpoint;
$ApnsCredential.ApnsCertificate = $ApnsCertificate;
$ApnsCredential.CertificateKey = $ApnsCertificateKey;
# Query the namespace
$FoundNamespaces = Get-AzureRmNotificationHubsNamespace -Namespace $HubNamespace -ResourceGroup $ResourceGroup
# Check if the namespace already exists
if ($FoundNamespaces -and $FoundNamespaces.Length -eq 1)
{
$CurrentNamespace = $FoundNamespaces[0];
Write-Host "The namespace [$HubNamespace] in the [$($CurrentNamespace.Location)] region was found.";
$HubListKeys = Get-AzureRmNotificationHubListKeys -Namespace $HubNamespace -ResourceGroup $ResourceGroup -NotificationHub $HubName -AuthorizationRule DefaultFullSharedAccessSignature;
# Check to see if the Notification Hub exists
if ($HubListKeys)
{
# Create the NamespaceManager object used to update the notification hub
Write-Host "Creating a NamespaceManager object for the [$HubNamespace] namespace...";
$NamespaceManager = [Microsoft.Azure.NotificationHubs.NamespaceManager]::CreateFromConnectionString($HubListKeys.PrimaryConnectionString);
Write-Host "NamespaceManager object for the [$HubNamespace] namespace has been successfully created.";
# Update notification hub with new details
Write-Host "The [$Path] notification hub already exists in the [$HubNamespace] namespace." ;
$NHDescription = $NamespaceManager.GetNotificationHub($HubName);
$NHDescription.GcmCredential = $GcmCredential;
$NHDescription.ApnsCredential = $ApnsCredential;
$NHDescription = $NamespaceManager.UpdateNotificationHub($NHDescription);
Write-Host "The [$HubName] notification hub was updated";
}
else
{
Write-Host "The [$HubName] notification hub does not exist."
}
}
else
{
Write-Host "The [$HubNamespace] namespace does not exist."
}
}
catch [System.Exception]
{
Write-Error($_.Exception.Message)
}
Hope that helps someone.
Is there a way to add a Hybrid Connection to a Azure Web App in an automated way? (either through PowerShell or a Resource Template or even a REST call to the Azure API?)
At the moment I have an Azure Web App being deployed with a Resource Manager Template and everything is being configured correctly, however I'm unable to figure out how I can link the Web App to an existing BizTalk Hybrid Connection in the template or via PowerShell (some sort of automated way).
The Azure documentation states that linking an Azure Web App to an existing Hybrid Connection can only be done manually via the Portal:
NOTE: The Web Apps portion of the Hybrid Connections feature is available only in the Azure Portal.
You can do this manually through the resource manager (https://resources.azure.com/) via a restful "Create" call when you drill down through your:
subscription -> resource groups -> sites -> -> hybridconnection
I have copied the call the resource manager app makes and replicated it in Powershell.
You need to:
Find your subscriptions tenant id (google it)
Register an application with AzureAD
Get that applications Id and secret
Collect various constants from resource manager such as a Biztalk Uri and something called an Entity Connection String
The script uses these details to get an auth token to call the rest api from powershell. It then calls the classic rest api to add the exisitng hybrid connection to the website. Note that this will only work on a website that has previously had a hybrid connection, so you cant spin up a brand new environment without some manual work and recording details from resource manager.
Below is a copy of this script, I call it from Octopus deploy so all #{...} variables are supplied from there.
This script will keep calling an api endpoint to the external facing website that calls an internal system via the hybrid connection. It will try 5 times waiting for a 200 response.
The Hybrid.ConnectionRestUrl variable I use in the script was acquired by watching the call the resource manager makes. It is constructed like this: https://management.azure.com/subscriptions/#{SubscriptionId}/resourceGroups/#{resource-group-name}/providers/Microsoft.Web/sites/#{web-site-name}/hybridconnection/#{web-site-name}?api-version=2015-08-01
No guarantee how long this script will work as it is hardly a supported method.
$authUri = "https://login.microsoftonline.com/#{tenant-domain}/oauth2/token"
$authMethod = "POST"
$authFormFields = #{resource='https://management.core.windows.net/';client_id='#{AzureAD.ApplicationId}';grant_type='client_credentials';client_secret='#{AzureAD.ApplicationSecret}'}
$authResponse = Invoke-WebRequest -Uri $authUri -Method $authMethod -Body $authFormFields -ContentType "application/x-www-form-urlencoded" | ConvertFrom-Json
$authorization = "Bearer " + $authResponse.access_token
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Authorization", $authorization)
$headers.Add("Content-Type", "application/json")
$URI = "#{Hybrid.ConnectionRestUrl}"
$method = "PUT"
$body = '{ "name" : "#{ExternalAzureService.WebApp}", "location" : "#{App.Location}", "type" : "Microsoft.Web/sites", "properties" : { "entityName" : "#{Hybrid.EntityName}", "entityConnectionString" : "#{Hybrid.EntityConnectionString}", "resourceType" : "", "resourceConnectionString" : "", "hostname" : "#{InternalService.Hostname.Raw}", "port" : 80, "biztalkUri" : "#{Hybrid.BiztalkUri}" } }'
Write-Output $URI
Write-Output $body
Try
{
$result = Invoke-RestMethod -Uri $URI -Method $method -Headers $headers -Body $body
}
Catch
{
Write-Output "Error Occurred "
$i = 1
$pingUrl = "http://#{ExternalAzureService.WebApp.HostName}/api/callinternalsystem"
Write-Output "Ping $i times this url: $pingUrl"
Do
{
Write-Output "Starting Ping call $i"
$response = Invoke-WebRequest $pingUrl
If ($response.StatusCode -eq 200) {
Write-Output "200 returned"
Break
}
$i++
} While ($i -le 5)
}
Write-Output " *********************************** SUCCESS ********************************** "
Write-Output $result
This can now be done in an Azure Resource Manager (ARM) template by using the Microsoft.Web sites/hybridConnectionNamespaces/relays template reference.
Here is an abbreviated example of what this could look like.
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"resources": [
{
"type": "Microsoft.Web/sites",
"apiVersion": "2018-11-01",
"resources": [
{
"apiVersion": "2019-08-01",
"dependsOn": [
"[resourceId('Microsoft.Web/sites', variables('appName'))]"
],
"name": "[concat(variables('relayName'), '/', parameters('hybridConnectionName'))]",
"properties": {
"hostname": "[split(json(reference(variables('hybridConnectionResourceId'), '2017-04-01').userMetadata)[0].value, ':')[0]]",
"port": "[split(json(reference(variables('hybridConnectionResourceId'), '2017-04-01').userMetadata)[0].value, ':')[1]]",
"relayArmUri": "[variables('hybridConnectionResourceId')]",
"relayName": "[parameters('hybridConnectionName')]",
"sendKeyName": "[parameters('hybridConnectionSendKeyName')]",
"sendKeyValue": "[listkeys(concat(variables('hybridConnectionResourceId'), '/authorizationRules/defaultSender'), '2017-04-01').primaryKey]",
"serviceBusNamespace": "[variables('relayName')]"
},
"tags": {
"displayName": "hc-some-name"
},
"type": "hybridConnectionNamespaces/relays"
}
]
}
]
}
The REST API for managing biztalk service can be found here
A reference to more developer resources (including powershell) https://msdn.microsoft.com/en-us/library/azure/dn832182.aspx.
I don't have an example of an ARM template, but maybe a schema could help you
https://github.com/Azure/azure-resource-manager-schemas/blob/master/schemas/2014-04-01/Microsoft.BizTalkServices.json