I am trying to update PowerBI datasource using Powershell in Azure devops Pipeline. This works sometimes and sometimes I get an error :
##[error]System.Threading.Tasks.TaskCanceledException: A task was canceled.
The command on which I get this error is :
$urlUpdateDatasource = [string]::Format("groups/{0}/datasets/{1}/Default.UpdateDatasources", $GroupID, $destinationDataset.Id);
$body= "{
""updateDetails"": [
{
""datasourceSelector"": {
""datasourceType"": ""$srcDBType"",
""connectionDetails"": {
""server"": ""$srcServer"",
""database"": ""$srcDB""
}
},
""connectionDetails"": {
""server"": ""$destinationServer"",
""database"": ""$destinationDB""
}
}
]
}"
Write-Host $body
Write-Host $urlUpdateDatasource
try{
$updateDS = Invoke-PowerBIRestMethod -Url $urlUpdateDatasource -Method Post -Body $body
}
catch{
Write-Host 'Failed to UpdateDatasources $_'
}
I have observed that when this step takes for than 1.5 min. I get this error though the update is successful . SO this error is false and shows my pipeline as failed even though it is not. Please suggest how I can handle this.
Related
When I retry unsuccessful operation using Powershell Durable Functions, it throws this exception:
HTTP Trigger script:
using namespace System.Net
param($Request, $TriggerMetadata)
$Request
$FunctionName = $Request.Params.FunctionName
$body = $Request.Body
$InstanceId = Start-NewOrchestration -FunctionName $FunctionName -Input $body
Orchestrator:
param($Context)
$data = $Context.input
Invoke-ActivityFunction -FunctionName 'O365_Holds_Activity' -Input $data
Activity Trigger:
using namespace System.Net
param([System.Collections.Hashtable] $message)
...
And function.json
{
"bindings": [
{
"name": "message",
"type": "activityTrigger",
"direction": "in"
}
],
"retry": {
"strategy": "fixedDelay",
"maxRetryCount": 3,
"delayInterval": "00:00:5"
}
}
First run has no problem with input, but the second one throws such exception. Btw, http request example:
HTTP POST: http://localhost:7071/api/orchestrators/O365_Orchestrator
Body:
{
"holdIds": [
1
],
"action": 0
}
In Durable orchestrators, I would recommend using the Durable-specific retry feature instead of the generic retry policies preview feature, for example:
$retryOptions = New-DurableRetryOptions `
-FirstRetryInterval (New-Timespan -Seconds 5) `
-MaxNumberOfAttempts 3
Invoke-DurableActivity -FunctionName $FunctionName -RetryOptions $retryOptions
The advantage of this approach is that the orchestration state gets persisted between retries, which eliminates the risk of the orchestrator invocation timing out because of retries, and make your orchestration more robust in general.
I want to analyze the log of the Ansible task inside a release pipeline with a custom task extension (node). In generell this is no big thing, but I ask my self whether it is possible to get the position of my executing task inside the pipeline?
Usecase
The user could add an Ansible task and if he want he can add my custom task to analyze the log (and publish somewhere). To decide which Ansible task inside the release pipeline should analyzed, the user should but the custom task directly after the Ansible task.
Something like this:
Release Pipeline 1.) Stage:
- Initialize job
- Download artifacts
- CopyFilesOverSSH
- Ansible <--- this should analyzed
- Custom Task
- Ansible
- Finialize job
Szenario
A release Pipeline with 3 Stages.
1.) Stage linear
2.) Stage & 3.) Stage parallel on second level
all stages contains a ansible task
all stages should contain my custom task extension to analyze the ansible task log
The Problem
When I now request the Get Release REST call I can loop though all environments, all jobs, all tasks looks like:
let release: Release = await Api.getRelease(Env.System.TeamProject, Env.Release.ReleaseId);
// loop environments (stages)
for (let environment of release.environments) {
// loop deploy steps
for (let deployStep of environment.deploySteps || []) {
// loop phases
for (let releaseDeployPhase of deployStep.releaseDeployPhases || []) {
// loop jobs
for (let deploymentJob of releaseDeployPhase.deploymentJobs || []) {
let ansible: ReleaseTask[] = [];
// loop tasks
for (let task of deploymentJob.tasks || []) {
if (task.startTime && task.finishTime) {
if (task.name === "Ansible") ansible.push(task);
}
}
console.log(ansible);
}
}
}
}
on runtime on this state my executing custom task is not finished yet (TaskStatus.InProgress), so together with the id of my task I am should able to detect the position inside the pipeline.
Some better Solution?
But I hope there is a much better solution, I mean the task should know which position it has inside the pipeline or? Maybe there is an information you can get over the azure-pipelines-task-lib/task lib?
Something like:
task.getRank() >>> 3
task.getEnvironment >>> 2
What also could help is when you could request the task name and the id inside the task program. Currently I have to create own variables with these information.
Something like this:
me.Id() >>> "8bb50e0a-8efb-47a4-b12a-2190f3a4d16a"
me.Name() >>> "MyCustomTask"
We could analyze the log of the Ansible task via power shell and REST API, an example is shown below:
$listurl="https://vsrm.dev.azure.com/{Org name}/{Project name}/_apis/release/releases/{Release id}?api-version=6.1-preview.8"
$PAT="{PAT}"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($PAT)"))
$result = Invoke-RestMethod -Uri $listurl -Headers #{Authorization = "Basic {0}" -f $base64AuthInfo} -Method get
$result.environments.deploySteps.releaseDeployPhases.deploymentJobs.tasks.name
foreach($deploySteps in $result.environments.deploySteps)
{
write-host $deploySteps.releaseDeployPhases.deploymentJobs.tasks.name
foreach($Task in $deploySteps.releaseDeployPhases.deploymentJobs.tasks){
if($Task.name -eq "{analyze task display name}")
{
#write-host $Task.logUrl
$TaskLogURL = $Task.logUrl
}
}
}
#write-host $TaskLogURL
$TaskLog = Invoke-RestMethod -Uri $TaskLogURL -Headers #{Authorization = "Basic {0}" -f $base64AuthInfo} -Method get
write-host $TaskLog
I have Azure function written in powershell and processing seems to be stopping for no apparent reason with timeout. Message never gets even to start being processed. This does not seem to be code related since code never even have a change to start since the first line of code is Write-Host "PowerShell queue trigger function processed work item: $QueueItem" which does not appear in log
Host.json
{
"version": "2.0",
"functionTimeout": "00:04:00",
"managedDependency": {
"enabled": true
},
"extensionBundle": {
"id": "Microsoft.Azure.Functions.ExtensionBundle",
"version": "[1.*, 2.0.0)"
},
"extensions": {
"queues": {
"maxPollingInterval": "00:00:02",
"visibilityTimeout": "00:00:30",
"batchSize": 16,
"maxDequeueCount": 2,
"newBatchThreshold": 8
}
}
}
Function.json
{
"bindings": [
{
"name": "QueueItem",
"type": "queueTrigger",
"direction": "in",
"queueName": "metadataservicequeue",
"connection": "useastbootdiag_STORAGE"
}
]
}
Script
# Input bindings are passed in via param block.
param([string] $QueueItem, $TriggerMetadata)
# Write out the queue message and insertion time to the information log.
Write-Host "PowerShell queue trigger function processed work item: $QueueItem"
Write-Host "Queue item insertion time: $($TriggerMetadata.InsertionTime)"
Write-Host "Starting executing Invoke-AzureRMCommand with parameters $($TriggerMetadata["VMName"]), $($TriggerMetadata["ResourceGroup"])"
$return = Invoke-AzureCommand -vmName $TriggerMetadata["VMName"] -resourceGroup $TriggerMetadata["ResourceGroup"]
Write-Host "Finished executing Invoke-AzureRMCommand with parameters $($TriggerMetadata["VMName"]), $($TriggerMetadata["ResourceGroup"])"
$json = #"
[
{
"Return" : $($return | convertto-json),
"VMName" : "$($TriggerMetadata["VMName"])",
"ResourceGroup" : "$($TriggerMetadata["ResourceGroup"])"
}
]
"#
Write-Host "Outputing following to Log Analytics $json"
Post-LogAnalyticsData -customerId $env:CustomerID -sharedKey $env:SharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($json)) -logType "MetaDataLog"
Powershell Azure Functions by default are not scaling well on consumption plan and poorely scale on dedicated plan due to default settings. If you function is not CPU bound and expected to scale well due to high number of queue items. You'd need to modify FUNCTIONS_WORKER_PROCESS_COUNT (https://learn.microsoft.com/en-us/azure/azure-functions/functions-app-settings) to higher number then default 1 and increase value of PSWorkerInProcConcurrencyUpperBound (https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference-powershell#concurrency) to higher number of default 1
I'm trying to loop through a JSON array of desired registry values, and then inspect the registry value for the correct setting.
The issue I have is that I'm not correctly defining the 'if..()' logic test in my loop. The problem code is located in the line: if($protocols[$i][$tempdisabledKVString] -eq "true")
I have the following object:
$protocolsJSON = #"
[
{
"Name": "TLS 1.2",
"Server-Enabled": True,
"Client-Enabled": True
}
]
"#
$protocols = $protocolsJSON | ConvertFrom-Json
Which fails the nested if statement below (undesired behavior)
elseif ($isDefaultDisabled -eq 0) # Protocol is manually enabled in registry (part 1.A)
{
if($protocols[$i][$tempdisabledKVString] -eq "True") # Protocol should be manually enabled in registry (part 1.B)
{
# For TLS 1.2 to be enabled and negotiated on servers that run Windows Server 2008 R2,
# you MUST create the DisabledByDefault entry in the appropriate subkey (Client, Server)
# and set it to "0". The entry will not be seen in the registry and it is set to "1" by default.
$errorString = "Warning: Protocol is only partially enabled."
$TLSProtocolResult.Errors = $errorString
}
else
{
write-host "DEBUG " $protocols[$i][$tempdisabledKVString]
write-host "DEBUG " $protocols[$i]
write-host "DEBUG " [$tempdisabledKVString]
$errorString = "Error: Protocol should be disabled."
$TLSProtocolResult.Errors = $errorString
}
}
Which produces the following output
DEBUG
DEBUG #{Name=TLS 1.2; Server-Enabled=True; Client-Enabled=True}
DEBUG [Server-Disabled]
DEBUG
DEBUG #{Name=TLS 1.2; Server-Enabled=True; Client-Enabled=True}
DEBUG [Client-Disabled]
How do I edit the IF statement so that I can test the true/false status of $protocols[$i][$tempdisabledKVString]?
The problem is you're trying to access a property as if it were a nested array.
Try this:
$protocolsJSON = #"
[
{
"Name": "TLS 1.2",
"Server-Enabled": true,
"Client-Enabled": true
}
]
"#
$protocols = $protocolsJSON | ConvertFrom-Json
$property = "Server-Enabled"
Write-Host "RESULT: $($protocols[0].$property)"
Your issue is most likely the JSON not being parsed. Try including quotes around your values. i.e. replace: "Server-Enabled": True, with "Server-Enabled": "True",.
Also, when if $tempdisabledKVString is the name of a property, you need to access it as a property rather than an index. i.e. replace $protocols[$i][$tempdisabledKVString] with $protocols[$i]."$tempdisabledKVString".
Clear-Host
$protocolsJSON = #"
[
{
"Name": "TLS 1.2",
"Server-Enabled": "True",
"Client-Enabled": "True"
}
]
"#
$protocols = $protocolsJSON | ConvertFrom-Json
$i = 0
$tempdisabledKVString = 'Server-Enabled'
if ($protocols[$i]."$tempdisabledKVString" -eq 'True') {
":)"
} else {
":("
}
In theory these issues should have caused exceptions to be thrown. For some reason you're not seeing those, or that would have prompted you to find the cause. Please check the value of $ErrorActionPreference; by default it should be set to Continue; but it looks like it may have been updated to SilentlyContinue for your session. It's OK to have this setting in some scenarios; but generally better to have errors be thrown when they occur so that you can see what's going wrong.
Is there a way to add a Hybrid Connection to a Azure Web App in an automated way? (either through PowerShell or a Resource Template or even a REST call to the Azure API?)
At the moment I have an Azure Web App being deployed with a Resource Manager Template and everything is being configured correctly, however I'm unable to figure out how I can link the Web App to an existing BizTalk Hybrid Connection in the template or via PowerShell (some sort of automated way).
The Azure documentation states that linking an Azure Web App to an existing Hybrid Connection can only be done manually via the Portal:
NOTE: The Web Apps portion of the Hybrid Connections feature is available only in the Azure Portal.
You can do this manually through the resource manager (https://resources.azure.com/) via a restful "Create" call when you drill down through your:
subscription -> resource groups -> sites -> -> hybridconnection
I have copied the call the resource manager app makes and replicated it in Powershell.
You need to:
Find your subscriptions tenant id (google it)
Register an application with AzureAD
Get that applications Id and secret
Collect various constants from resource manager such as a Biztalk Uri and something called an Entity Connection String
The script uses these details to get an auth token to call the rest api from powershell. It then calls the classic rest api to add the exisitng hybrid connection to the website. Note that this will only work on a website that has previously had a hybrid connection, so you cant spin up a brand new environment without some manual work and recording details from resource manager.
Below is a copy of this script, I call it from Octopus deploy so all #{...} variables are supplied from there.
This script will keep calling an api endpoint to the external facing website that calls an internal system via the hybrid connection. It will try 5 times waiting for a 200 response.
The Hybrid.ConnectionRestUrl variable I use in the script was acquired by watching the call the resource manager makes. It is constructed like this: https://management.azure.com/subscriptions/#{SubscriptionId}/resourceGroups/#{resource-group-name}/providers/Microsoft.Web/sites/#{web-site-name}/hybridconnection/#{web-site-name}?api-version=2015-08-01
No guarantee how long this script will work as it is hardly a supported method.
$authUri = "https://login.microsoftonline.com/#{tenant-domain}/oauth2/token"
$authMethod = "POST"
$authFormFields = #{resource='https://management.core.windows.net/';client_id='#{AzureAD.ApplicationId}';grant_type='client_credentials';client_secret='#{AzureAD.ApplicationSecret}'}
$authResponse = Invoke-WebRequest -Uri $authUri -Method $authMethod -Body $authFormFields -ContentType "application/x-www-form-urlencoded" | ConvertFrom-Json
$authorization = "Bearer " + $authResponse.access_token
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Authorization", $authorization)
$headers.Add("Content-Type", "application/json")
$URI = "#{Hybrid.ConnectionRestUrl}"
$method = "PUT"
$body = '{ "name" : "#{ExternalAzureService.WebApp}", "location" : "#{App.Location}", "type" : "Microsoft.Web/sites", "properties" : { "entityName" : "#{Hybrid.EntityName}", "entityConnectionString" : "#{Hybrid.EntityConnectionString}", "resourceType" : "", "resourceConnectionString" : "", "hostname" : "#{InternalService.Hostname.Raw}", "port" : 80, "biztalkUri" : "#{Hybrid.BiztalkUri}" } }'
Write-Output $URI
Write-Output $body
Try
{
$result = Invoke-RestMethod -Uri $URI -Method $method -Headers $headers -Body $body
}
Catch
{
Write-Output "Error Occurred "
$i = 1
$pingUrl = "http://#{ExternalAzureService.WebApp.HostName}/api/callinternalsystem"
Write-Output "Ping $i times this url: $pingUrl"
Do
{
Write-Output "Starting Ping call $i"
$response = Invoke-WebRequest $pingUrl
If ($response.StatusCode -eq 200) {
Write-Output "200 returned"
Break
}
$i++
} While ($i -le 5)
}
Write-Output " *********************************** SUCCESS ********************************** "
Write-Output $result
This can now be done in an Azure Resource Manager (ARM) template by using the Microsoft.Web sites/hybridConnectionNamespaces/relays template reference.
Here is an abbreviated example of what this could look like.
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"resources": [
{
"type": "Microsoft.Web/sites",
"apiVersion": "2018-11-01",
"resources": [
{
"apiVersion": "2019-08-01",
"dependsOn": [
"[resourceId('Microsoft.Web/sites', variables('appName'))]"
],
"name": "[concat(variables('relayName'), '/', parameters('hybridConnectionName'))]",
"properties": {
"hostname": "[split(json(reference(variables('hybridConnectionResourceId'), '2017-04-01').userMetadata)[0].value, ':')[0]]",
"port": "[split(json(reference(variables('hybridConnectionResourceId'), '2017-04-01').userMetadata)[0].value, ':')[1]]",
"relayArmUri": "[variables('hybridConnectionResourceId')]",
"relayName": "[parameters('hybridConnectionName')]",
"sendKeyName": "[parameters('hybridConnectionSendKeyName')]",
"sendKeyValue": "[listkeys(concat(variables('hybridConnectionResourceId'), '/authorizationRules/defaultSender'), '2017-04-01').primaryKey]",
"serviceBusNamespace": "[variables('relayName')]"
},
"tags": {
"displayName": "hc-some-name"
},
"type": "hybridConnectionNamespaces/relays"
}
]
}
]
}
The REST API for managing biztalk service can be found here
A reference to more developer resources (including powershell) https://msdn.microsoft.com/en-us/library/azure/dn832182.aspx.
I don't have an example of an ARM template, but maybe a schema could help you
https://github.com/Azure/azure-resource-manager-schemas/blob/master/schemas/2014-04-01/Microsoft.BizTalkServices.json