How to stop a Storage Event Trigger of Azure Data Factory using Powershell when there is a Delete lock on the Resource group? - azure-devops

I want to stop a Storage Event Trigger that is on my data factory before I make modifications to the factory using ARM deployment/Azure DevOps. There is a Delete lock on my resource group which is causing the below error when I try to stop the trigger using powershell (Stop-AzDataFactoryV2Trigger) :
Error Code: BadRequest
Error Message: The scope '/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Storage/storageAccounts/XXX/providers/Microsoft.EventGrid/eventSubscriptions/XXX'
cannot perform delete operation because following scope(s) are locked: '/subscriptions/XXX/resourceGroups/XXX'. Please remove the lock and try again.
Is there any way to do my ADF deployments without having to remove this Delete lock?

After a bit of research and digging around, I found out that the direct answer to this question is that it's not possible to Start/Stop a Storage Event Trigger on a Data Factory when there is a Delete lock on the entire Resource Group. This is because whenever a Storage Event Trigger is started or stopped, an Event Subscription (which is a resource) is created and deleted in the Resource Group but with a Delete lock in place, this deletion cannot happen.
However, there are few workarounds to address this requirement :
Have a Delete lock at the Resource level and not at the Resource
Group level.
Move the Data Factory and the Storage Account to a different Resource Group
which doesn't have a Delete lock.
Delete the "Delete lock" before the deployment of the ADF and recreate it after
the deployment. For this, the Service Principal being used to do the deployments
should have the permission needed to update/delete locks.
If anyone has a direct solution to this problem, I'm happy to accept that as the answer. Thanks.

The following sample script can be used to stop triggers before deployment.
if ($predeployment -eq $true) {
#Stop all triggers
Write-Host "Stopping deployed triggers`n"
$triggersToStop | ForEach-Object {
if ($_.TriggerType -eq "BlobEventsTrigger" -or $_.TriggerType -eq "CustomEventsTrigger") {
Write-Host "Unsubscribing" $_.Name "from events"
$status = Remove-AzDataFactoryV2TriggerSubscription -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name
while ($status.Status -ne "Disabled"){
Start-Sleep -s 15
$status = Get-AzDataFactoryV2TriggerSubscriptionStatus -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name
}
}
Write-Host "Stopping trigger" $_.Name
Stop-AzDataFactoryV2Trigger -ResourceGroupName $ResourceGroupName -DataFactoryName $DataFactoryName -Name $_.Name -Force
}
For more information follow this Pre- and Post-deployment script given in official documentation.

Related

Unable to update Azure Cloud Service Extended Support via Azure Powershell Task

I am attempting to update Azure Cloud Service Extended Support using Azure PowerShell task in Azure DevOps Release pipeline. I receive the follow error when using Update-AzCloudService:
Specified api-version 2021-03-01 does not meet the minimum required api-version 2022-04-04 to set this property SlotType
Here is the code block used in the PowerShell task:
...
$cloudServiceIdentity = #{
CloudServiceName = $cloudServiceName
Location = $location
ResourceGroupName = $resourceGroupName
SubscriptionId = $subscription.Subscription.Id
}
Write-Host "-------------------------------"
Write-Host = "cloudServiceIdentity.CloudServiceName= " $cloudServiceIdentity.CloudServiceName
Write-Host = "cloudServiceIdentity.Location = " $cloudServiceIdentity.Location
Write-Host = "cloudServiceIdentity.ResourceGroupName = " $cloudServiceIdentity.ResourceGroupName
Write-Host = "cloudServiceIdentity.SubscriptionId = " $cloudServiceIdentity.SubscriptionId
Write-Host "-------------------------------"
$cloudService = #{
Location = $location
configurationUrl = $configSas
PackageUrl = $packageSas
OSProfile = $osProfile
}
Write-Host "-------------------------------"
Write-Host = "cloudService.Location= " $location
Write-Host = "cloudService.configurationUrl = " $configSas
Write-Host = "cloudService.PackageUrl = " $packageSas
Write-Host = "cloudService.OSProfile = " $osProfile
Write-Host "-------------------------------"
Write-Host "Starting Azure CSES Update"
Update-AzCloudService -InputObject $cloudServiceIdentity -Parameter $cloudService
Write-Host "Completed Azure CSES Update"
...
I have confirmed that Az.CloudService is running version is 1.1.0 (latest version)
Environment info:
| Environment | Value |
|:----------- |:------|
|Azure DevOps Version|18.181.31626.1 (Azure DevOps Server 2020 Update 1.1)|
|Azure DevOps Azure PowerShell Task |Version 5|
|Azure PowerShell version options - Preferred Azure PowerShell Version| 3.1.0|
|Az.CloudService | 1.1.0|
I have been struggling with this issue for several days and cannot seem to find any information on fixing the problem.
Any insights are greatly appreciated!
Given the error message you shared, I suppose that you could check this comment about api version error.
And run the command below to check the latest released api version.
((get-azresourceprovider -providernamespace Microsoft.Compute).resourcetypes | ?{$_.ResourceTypeName -eq "virtualmachines"}).apiversions
And remodify your api-version parameter in your json or bicep template.
The issue does appear to be related to the AzCloudService Powershell Module or the CloudService Resource by itself. When creating the resource, Azure does seem to select the 2021-03-01 API version even if you have a newer version selected in your ARM Template.
It could be that this tight versioning requirements with the way Powershell makes its queries to the resource doesn't seem to work well with the resource.
In my case, I was using the New-AzCloudService cmdlet and always encountered the issue (note, to update the CloudService resource, you may still need to use the New-AzCloudService cmdlet. If the resource exists, an update operation will be performed. If the resource doesn't exist, a new resource will be created).
I noticed though that if using Powershell, all the parameters need to be specified if using the New-AzCloudService cmdlet.
As specified here in the example Step 13:
https://learn.microsoft.com/en-us/azure/cloud-services-extended-support/deploy-powershell#create-cloud-service-using-profile-objects--sas-uris
you must have to pass in the other parameters before your query will be accepted (at least this is what worked for me). -Tag was not required.
So initially I was passing in my query as follows:
$cloudService = New-AzCloudService `
-Name “ContosoCS” `
-ResourceGroupName “ContosOrg” `
-Location “East US” `
-PackageUrl $cspkgUrl `
-ConfigurationUrl $cscfgUrl
because I only wanted to perform a release update and I always got the error:
Specified api-version 2021-03-01 does not meet the minimum required api-version 2022-04-04 to set this property SlotType
When I passed in all other parameters (I referenced the existing cloud service's parameters since I'm performing an in-place update), it got accepted.
$cloudService = New-AzCloudService `
-Name “ContosoCS” `
-ResourceGroupName “ContosOrg” `
-Location “East US” `
-PackageUrl $cspkgUrl `
-ConfigurationUrl $cscfgUrl `
-UpgradeMode 'Auto' `
-RoleProfile $roleProfile `
-NetworkProfile $networkProfile `
-ExtensionProfile $extensionProfile `
-OSProfile $osProfile
Hope this helps.

Check whether a Cosmos DB exists using PowerShell

I am trying to check whether a Cosmos DB exists or not using Powershell in Octopus. If Not I need to create it. Thats the requirement
$ApplicationShortName = "stc"
$resourceGroup = $OctopusParameters["AzurePlatform.Application[$ApplicationShortName].ResourceGroup.Name"]
$CosmosAccount = $OctopusParameters["AzurePlatform.Application[$ApplicationShortName].CosmosDbAccount.Name"]
$databaseName='sdsd'
Write-Host "Resource Group : $resourceGroup"
Write-host "Cosmos Account : $CosmosAccount"
#Check whether database exists
Get-AzCosmosDBSqlDatabase -ResourceGroupName $resourceGroup -AccountName $CosmosAccount -Name $databaseName
But here the problem is if DB actually exists, the above function works fine. But if DB not exists, it simply triggers an error.
So how to check whether the DB exists or not. So if not exists I need to fire this command
New-AzCosmosDBSqlDatabase -AccountName $CosmosAccount -Name $databaseName -ResourceGroupName $resourceGroup
You should be able to handle this process within your Octopus script - take a look at the documentation for error handling for Octopus scripts
The best practice here is to always check the exit code when invoking programs:
& ping 255.255.255.0
if ($LastExitCode -ne 0) {
throw "Couldn't find 255.255.255.0"
}
By checking the $LastExitCode, you can determine whether there was a success in your step and drive your database creation.
Also worth noting that if you use the az cli instead of the PowerShell commands, there's a specific command for what you're doing here that returns a boolean! Check out az cosmosdb database exists if you want to try and get it that way without having to manually check exit codes.

TFS 2017 How to rollback a release

Has anyone has a robust way to rollback if a release fails? The methods mentioned in this article doesn't sound very practical: https://blogs.msdn.microsoft.com/devops/2016/03/28/implement-rollback-with-release-management-for-tfs-2015/
IBM UrbanCode Deploy can choose to redeploy the previous successful deployment w/o any user intervention: https://developer.ibm.com/urbancode/videos/rollback-scenarios-in-ibm-urbancode-deploy/
Can we have that in TFS?
Once you install the Release Management Utility Tasks extension in your account, you’ll see a task called “Rollback powershell” in the task catalog. For rollback, you just need to add this task to the release definition and mark it “always run”.
An example to access the task execution information is as follows.
try
{
$jsonobject = ConvertFrom-Json $env:Release_Tasks
}
catch
{
Write-Verbose -Verbose “Error parsing Release_Tasks environment variable”
Write-Verbose -Verbose $Error
}
foreach ($task in $jsonobject | Get-Member -MemberType NoteProperty)
{
$taskproperty = $jsonobject.$($task.Name) | ConvertFrom-Json
Write-Verbose -Verbose “Task $($taskproperty.Name) with rank $($task.Name) has status $($taskproperty.Status)”
// Perform rollback action required in case $task.Name has status failed
}
Please notice, you may need to update the api version. Check the case below:
https://social.msdn.microsoft.com/Forums/expression/en-US/aacab4c3-b25b-4348-90b1-4d5661d9d148/release-management-utility-tasks-rollback-task-is-not-working-in-tfs-2017-update-1?forum=tfsbuild

Stop-AzureWebsiteJob results in "Not Found"

so I've set up a build/deployment in VSO where I want to stop the website and web jobs before deployment (because otherwise files are locked). The powershell script essentially contains:
Stop-AzureWebsite -Name $website # this works, website is stopped after
$jobs = Get-AzureWebsiteJob -Name $website # this works, contains a list of the jobs
Stop-AzureWebsiteJob -Name $website -JobName $job -PassThru
This last line fails, using the names returned from the preceding call I get an unhelpful "Not Found". It's not an account / subscription thing as the preceding lines work happily, does anyone have any ideas?
Thanks in advance
The solution is Remove-AzureWebsiteJob.
Regarding Stop-AzureWebsiteJob command, Starting and stopping Web Jobs is an operation that is only valid on continuous Web Jobs.
Based on the debug log (add -Debug to the command), it is trying to stop a continuous job https://[sitename].scm.azurewebsites.net/api/jobs/continuous/[jobname]/stop.
Regarding your requirement, you can try to stop the job through Kudu API.
There is the sample for using Azure App Service Kudu REST API that you can refer to:
Samples for using the Azure App Service Kudu REST API to programmatically manage files in your site
Related thread: Azure Stop a Triggered Web Job
Here is a simple command:
Invoke-AzResourceAction -ResourceGroupName <RG-NAME> -ResourceType Microsoft.Web/sites/continuouswebjobs -ResourceName <APP-SERVICE/WebJobName> -Action stop -ApiVersion 2018-02-01 -Force

How to list all running webjobs within an azure subscription using powershell

I have an azure subscription that has upwards of 200 appServices where around about half of them have Continuous, always on webJobs attached, some also have slots which also have webJobs.
Is there a way to list all webJobs that are inside a subscription? I was originally tring to use powershell to do this but it was getting rather complex and was wondering if anyone knew of an easy way to achieve the above.
It seems like Get-AzureRmWebApp should be able to help but i cant find a way to list the jobs that reside inside the webapps.
I found the command Get-AzureWebsiteJob which is not in the AzureRM family of commandlets. The following script can get the data that I'm looking for:
$groups = get-AzureRmResourceGroup | where{$_.ResourceGroupName -like "*-prod-*"}
foreach($group in $groups){
Write-Host -ForegroundColor Cyan "processing resourceGroup" $group.ResourceGroupName
$webApps = Get-AzureRmWebApp -ResourceGroupName $group.ResourceGroupName
foreach($webApp in $webApps){
write-host -ForegroundColor Yellow $webApp.Name
$job = Get-AzureWebsiteJob -Name $webApp.Name
if($job){
write-host -ForegroundColor DarkYellow $job.JobName
}
$job = Get-AzureWebsiteJob -Name $webApp.Name -Slot staging
if($job){
write-host -ForegroundColor DarkYellow $job.JobName " -staging"
}
}
}
The above does not filter out the running ones from the stopped, but that can be easily added if need be.
Of course you firstly need to be logged into AzureRM and Azure classic
Login-AzureRmAccount
Select-AzureRmSubscription -SubscriptionId <<mySubscriptionId>>
Get-AzureRmContext
Add-AzureAccount
Select-AzureSubscription -SubscriptionId <<mySubscriptionId>>
Get-AzureSubscription -Current
Its a very slow script iterating over this number or AppServices though. Any ideas for speeding it up would be appreciated.
You can do it through the ARM APIs, though you still need to call it on each Web App.
You can get the WebJobs with a GET request to:
https://management.azure.com/subscriptions/subscription-id/resourceGroups/resource-group-name/providers/Microsoft.Web/sites/app-name/webjobs?api-version=2016-03-01
But I doubt this will be any more efficient than what you have since you still need to make a call for every Web App. And you will need to get the access token somehow.
Web Jobs are a property of App Service applications and can't be requested all at once from Azure.
I know it's too late but we can try with Azure CLI too these days.
az webapp webjob triggered list --name MyWebApp --resource-group MyResourceGroup
Additionally, you can query the above result and get the desired information you are looking for
az webapp webjob triggered list --name MyWebApp --resource-group MyResourceGroup --query "[].{Name:name, Schedule:settings.schedule}"
I know it's too late but we can try with Azure CLI too these days.
az webapp webjob triggered list --name MyWebApp --resource-group MyResourceGroup
Additionally, you can query the above result and get the desired information you are looking for
az webapp webjob triggered list --name MyWebApp --resource-group MyResourceGroup --query "[].{Name:name, Schedule:settings.schedule}"
But, again you have to run the above query in the for each loop to fetch the results from all resource groups in the subscription.