I am trying to create HDInsight cluster in Azure with Metastore using the Powershell script. But it is throwing BadRequest: RegionCapabilityNotAvailable,Region capability not available for region 'East US' error. But East US is a supported region for the HDInsight cluster. Please find my code below.
$storageAccountResourceGroupName = "hdi-rg"
$storageAccountName = "qwertyhdi"
#$storageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $storageAccountResourceGroupName -Name $storageAccountName)[0].value
$storageContainer = "qwertyiopasdf-2020-05-03t08-30-23-118z"
# Cluster configuration info
$location = "East US"
$clusterResourceGroupName = "hdi-rg"
$clusterName = "qwertyiopasdf"
$username = "admin"
$password = ConvertTo-SecureString "password" -AsPlainText -Force
$clusterCreds = New-Object System.Management.Automation.PSCredential -ArgumentList ($username, $password)
# Hive metastore info
$hiveSqlServer = "server"
$hiveDb = "db123"
$sqlusername = "qwerty"
$sqlpassword = ConvertTo-SecureString "password" -AsPlainText -Force
$hiveCreds = New-Object System.Management.Automation.PSCredential -ArgumentList ($sqlusername, $sqlpassword)
New-AzStorageAccount `
-ResourceGroupName $storageAccountResourceGroupName `
-Name $storageAccountName `
-Location $location `
-SkuName Standard_LRS `
-Kind StorageV2 `
-EnableHttpsTrafficOnly 1
# Note: Storage account kind BlobStorage cannot be used as primary storage.
$storageAccountKey = (Get-AzStorageAccountKey `
-ResourceGroupName $storageAccountResourceGroupName `
-Name $storageAccountName)[0].Value
$defaultStorageContext = New-AzStorageContext `
-StorageAccountName $storageAccountName `
-StorageAccountKey $storageAccountKey
New-AzStorageContainer `
-Name $storageContainer `
-Context $defaultStorageContext #use the cluster name as the container name
$metastore = New-AzHDInsightClusterConfig | Add-AzHDInsightMetastore -SqlAzureServerName "$hiveSqlServer.database.windows.net" -DatabaseName $hiveDb -Credential $hiveCreds -MetastoreType HiveMetastore
New-AzHDInsightCluster -Location $location -ResourceGroupName $clusterResourceGroupName -ClusterName $clusterName -ClusterType Hadoop -OSType Windows -ClusterSizeInNodes 1 -HttpCredential $clusterCreds -DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" -DefaultStorageAccountKey $storageAccountKey -DefaultStorageContainer $storageContainer -Config $metastore
Is -OSType Windows still valid. I realize "Windows" is listed as an option from the PowerShell specs, but I think "linux" is the only value that will actually work.
You will receive this error message BadRequest: RegionCapabilityNotAvailable,Region capability not available for region 'East US', when underlying compute sku is not available for the selected region in that subscription.
I would request you to check if the resource are available from Azure Portal.
Azure Portal => Select your subscription => Usage + Quotas
Filter with => Microsoft.Compute and Region => East US
If you're unable to find a suitable SKU in that region/zone or an alternative region/zone that meets your business needs, submit a SKU request or Quota increase to Azure Support.
How to check if Microsoft Azure KeyVault exist? I am creating with the following script via PowerShell a new KeyVault
$vaultName = "kv3"
Set-AzureRmContext -Subscription "yyyy-f48c-4e0c-80f4-cxxxx"
New-AzureRmKeyVault -VaultName $vaultName -ResourceGroupName 'bogl' -Location 'North Europe'
$secretvalue = ConvertTo-SecureString '4444' -AsPlainText -Force
$secret = Set-AzureKeyVaultSecret -VaultName $vaultName -Name 'mySecret' -SecretValue $secretvalue
--> Check if KeyVault was successfully created/deployment process was finished or wait ...
$status = Get-AzureRMKeyVault -VaultName $vaultName
Write-Host $status
When KeyVault deployment process takes a little bit longer and the KeyVault is not yet available, then i want to wait some seconds.
you can put a simple loop that will test whether the KV is created -
while (-not (Get-AzureRmKeyVault -VaultName "test-keyvault" ))
{
sleep 10
}
Azure Search does not give any scheduled scaling option so I try to make it by Automation account.
I followed AzSearch PowerShell command, but it does not work as I expected.
Set-AzureRmResource with ReplicaCount=2 parameter is not applied. Actually, it does not give any result message. What did I miss?
To reproduce my problem, you can import my runbook file at below link;
https://gist.github.com/YoungjaeKim/5cb66a666a3a864b7379aac0a400da40
Save the text file as AzureSearch-SetReplicaCount.graphrunbook and Import it to Automation account > Add a runbook menu.
Following commenters, I ended up by making PowerShell runbook.
I uploaded powershell source code to below link;
https://gallery.technet.microsoft.com/scriptcenter/Azure-Search-change-c0b49c4c
Let me attach the code as below;
<#
.DESCRIPTION
Scale Azure Search ReplicaCount
AzSearch command reference; https://learn.microsoft.com/en-us/azure/search/search-manage-powershell
.NOTES
AUTHOR: Youngjae Kim
LASTEDIT: June 19, 2017
#>
Param(
[string]$SubscriptionId,
[string]$ResourceGroupName,
[string]$AzSearchResourceName,
[int]$InstanceCount = 1
)
# 1. Acquire Automation account
$connectionName = "AzureRunAsConnection"
try
{
# Get the connection "AzureRunAsConnection "
$servicePrincipalConnection=Get-AutomationConnection -Name $connectionName
"Logging in to Azure..."
Add-AzureRmAccount `
-ServicePrincipal `
-TenantId $servicePrincipalConnection.TenantId `
-ApplicationId $servicePrincipalConnection.ApplicationId `
-CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint
}
catch {
if (!$servicePrincipalConnection)
{
$ErrorMessage = "Connection $connectionName not found. You must have Automation account. Reference: https://learn.microsoft.com/en-us/azure/automation/automation-role-based-access-control"
throw $ErrorMessage
} else{
Write-Error -Message $_.Exception
throw $_.Exception
}
}
# 2. Select subscription
Select-AzureRmSubscription -SubscriptionId $SubscriptionId
# 3. Specify Azure Search Resource
$resource = Get-AzureRmResource `
-ResourceType "Microsoft.Search/searchServices" `
-ResourceGroupName $ResourceGroupName `
-ResourceName $AzSearchResourceName `
-ApiVersion 2015-08-19
Write-Output ($resource)
# 4. Scale your service up
# Note that this will only work if you made a non "free" service
# This command will not return until the operation is finished
Write-Output ("Updating InstanceCount to " + $InstanceCount + ". This can take 15 minutes or more...")
$resource.Properties.ReplicaCount = $InstanceCount
$resource | Set-AzureRmResource -Force -Confirm:$false
# 5. Finish
Write-Output ("End of Process to set InstanceCount = " + $InstanceCount + " for " + $AzSearchResourceName)
I want to upscale and downscale my Azure Analysis Services with PowerShell (Automation Runbook), but changing the Tier (Sku) doesn't seem to work. However there are no errors. Any suggestions?
# PowerShell code
# Connect to a connection to get TenantId and SubscriptionId
$Connection = Get-AutomationConnection -Name "AzureRunAsConnection"
$TenantId = $Connection.TenantId
$SubscriptionId = $Connection.SubscriptionId
# Get the service principal credentials connected to the automation account.
$null = $SPCredential = Get-AutomationPSCredential -Name "SSISJoost"
# Login to Azure ($null is to prevent output, since Out-Null doesn't work in Azure)
Write-Output "Login to Azure using automation account 'SSISJoost'."
$null = Login-AzureRmAccount -TenantId $TenantId -SubscriptionId $SubscriptionId -Credential $SPCredential
# Select the correct subscription
Write-Output "Selecting subscription '$($SubscriptionId)'."
$null = Select-AzureRmSubscription -SubscriptionID $SubscriptionId
# Get variable values
$ResourceGroupName = Get-AutomationVariable -Name 'ResourceGroupName'
$AnalysisServerName = Get-AutomationVariable -Name 'AnalysisServerName'
# Get old status (for testing/logging purpose only)
$OldAsSetting = Get-AzureRmAnalysisServicesServer -ResourceGroupName $ResourceGroupName -Name $AnalysisServerName
try
{
# changing tier
Write-Output "Upgrade $($AnalysisServerName) to S1. Current tier: $($OldAsSetting.Sku.Name)"
Set-AzureRmAnalysisServicesServer -ResourceGroupName $ResourceGroupName -Name $AnalysisServerName -Sku "S1"
}
catch
{
Write-Error -Message $_.Exception
throw $_.Exception
}
Write-Output "Done"
# Get new status (for testing/logging purpose only)
$NewAsSetting = Get-AzureRmAnalysisServicesServer -ResourceGroupName $ResourceGroupName -Name $AnalysisServerName
Write-Output "New tier: $($NewAsSetting.Sku.Name)"
using Set-AzureRmAnalysisServicesServer
There was a little bug in the PowerShell AzureRM.AnalysisServices module. It has been fixed in 0.4.0 (Thursday, June 08 2017)
Now the code finally works: http://microsoft-bitools.blogspot.com/2017/06/schedule-upscaledownscale-azure.html
I am building a power shell script to automate the setup of a website environment in Azure. This web uses an account storage. I want to the script not to create the account storage if exists.
I thought that using Get-AzureStorageAccount this way may work but it does not:
Write-Verbose "[Start] creating $Name storage account $Location location"
$storageAcct = Get-AzureStorageAccount –StorageAccountName $Name
if (!$storageAcct)
{
$storageAcct = New-AzureStorageAccount -StorageAccountName $Name -Location $Location -Verbose
if ($storageAcct)
{
Write-Verbose "[Finish] creating $Name storage account in $Location location"
}
else
{
throw "Failed to create a Windows Azure storage account. Failure in New-AzureStorage.ps1"
}
}
else
{
Write-Verbose "$Name storage account in $Location location already exists, skipping creation"
}
The issue is I don't know how to handle the return of Get-AzureStorageAccount.
Thank you very much in advance!
I would suggest using the Test-AzureName cmdlet to determine if it exists. So, something like this.
if (!(Test-AzureName -Storage $Name))
{
Write-Host "Creating Storage Account $Name"
New-AzureStorageAccount -StorageAccountName $Name -Location $Location
}
You can use Test-AzureName for other services too, such as Cloud Services, WebSites, and ServiceBus. It returns True if it exists, False otherwise.
Get-AzureRmStorageAccountNameAvailability -Name "accountname"
Try this:
$Name = "myStorageAccount"
$Location = "myLocation"
Write-Host "[Start] creating $Name storage account $Location location"
try{
Get-AzureStorageAccount –StorageAccountName $Name -ErrorAction Stop | Out-Null
Write-Host "$Name storage account in $Location location already exists, skipping creation"
}
catch{
Write-Host "[Finish] creating $Name storage account in $Location location"
New-AzureStorageAccount -StorageAccountName $Name -Location $Location -Verbose
}
Test-AzureName didn't work with our build agents and we already had a try/catch in code so a second one would require building it out as a function. I opted for that standard get and check if null, use -ErrorAction Ignore to stop it throwing an exception
# Check for storage account and create if not found
$StorageAccount = Get-AzureRmStorageAccount -Name $StorageAccountName -ResourceGroupName $StorageAccountRG -ErrorAction Ignore
if ($StorageAccount -eq $null)
{
New-AzureRmStorageAccount -Location "West Europe" -Name $StorageAccountName -ResourceGroupName $StorageAccountRG -SkuName Standard_LRS -Kind Storage
$StorageAccount = Get-AzureRmStorageAccount -Name $StorageAccountName -ResourceGroupName $StorageAccountRG
}
#Rick Rainey's solution works if you're logged in using Add-AzureAccount. However, Azure and powershell have a conflicting and confusing suite of login accounts (Windows Live versus AD) and login mechanisms (Classic: Add-AzureAccount; Resource manager: Login-AzureRmAccount). Some Azure powershell cmdlets require a specific login; further, some require a specific account type!
To clear through this thicket of complicated, undocumented, and confusing permission issues, we always use an AD account, logging in via Login-AzureRmAccount. We also use Azure resource manager (ARM) resources and cmdlets, following Microsoft's movement to ARM as its recommended and strategic approach. However, #RIck's solution is one which the ARM login doesn't work with. :-( So you need another approach, which is #Darren's (for storage). However, for a generic replacement for Test-AzureName I'd suggest Find-AzureRmResource. In the case of storage
$StorageObject = Find-AzureRmResource -ResourceType "Microsoft.Storage/storageAccounts" | Where-Object {$_.Name -eq $storageName}
if ( !$StorageObject ) {
$storageLocation = (Get-AzureRmResourceGroup -ResourceGroupName $resourceGroup).Location
$storageType = "Standard_LRS"
New-AzureRmStorageAccount -ResourceGroupName $resourceGroup -Name $storageName -Location $storageLocation -Type $storageType
}
You should use the latest Powershell module Az.
if ($(Get-AzStorageAccount -ResourceGroupName $resourceGroupName -Name $storageAccountName) -eq $null)
{
# does not exist
}
With the current Az module for PowerShell Version 7, the Get-AzStorageAccountNameAvailability cmdlet might offer a more efficient solution as it was designed specifically for this task. Here is an example:
# ... declare variables and specify values ...
$checkNameAvail = (Get-AzStorageAccountNameAvailability -Name $storageAccountName) | `
Select-Object NameAvailable
if ($checkNameAvail.NameAvailable)
{
Write-Host 'Account name available! Please wait while your resource is being created'
# Create account. Variables used in this example would have been declared earlier in the script.
$storageAccount = (New-AzStorageAccount -ResourceGroupName $resourceGroupName `
-AccountName $storageAccountName `
-Location $location `
-SkuName $skuType `
-AllowBlobPublicAccess $false -EnableHttpsTrafficOnly $true)
# ...
}
else
{
# This section of the script executes if the name is not available
Write-Host "The name <$storageAccountName> is not available. Suggest a new globally unique name!"
}
The condition above will return False, and execute the else statement because the boolean value returned by the cmdlet is in [0] as shown in the PowerShell command-line test below. The availability information (boolean) can thus be stripped from the object returned by the cmdlet and (as in this example) used as a condition in the rest of the script.
PS C:\> Get-AzStorageAccountNameAvailability -Name testaccount1
NameAvailable Reason Message
------------- ------ -------
False AlreadyExists The storage account named testaccount1 is already taken.
Use the error variable
Get-AzStorageAccount -ResourceGroupName 'RG-QA-TEST' -Name 'staccountfor12334ff' -ErrorVariable ev1 -ErrorAction SilentlyContinue
if ($ev1) {
Write-Host "-------------------------- Creating OEM Storage"
//create storage account
}
I had this challenge when setting up Azure storage accounts for Static website hosting using Powershell in Octopus Deploy.
Here's how I fixed it:
Using the Az module for Azure Powershell I did the following:
# Define Variables
$RESOURCE_GROUP_NAME = my-resource-group
$LOCATION = northeurope
$STORAGE_ACCOUNT_NAME = myapplication
$SKU_NAME = Standard_GRS
$STORAGE_KIND = StorageV2
# Check Storage Account and Create if not Found
$STORAGE_ACCOUNT = Get-AzStorageAccount -ResourceGroupName $RESOURCE_GROUP_NAME -Name $STORAGE_ACCOUNT_NAME -ErrorAction Ignore
if ($STORAGE_ACCOUNT -eq $null) {
Write-Host 'Creating storage account'
New-AzStorageAccount -ResourceGroupName $RESOURCE_GROUP_NAME -AccountName $STORAGE_ACCOUNT_NAME -Location $LOCATION -SkuName $SKU_NAME -Kind $STORAGE_KIND
Write-Host "$STORAGE_ACCOUNT_NAME storage account successfully created"
}
else {
Write-Host "$STORAGE_ACCOUNT_NAME storage account already exists"
}
Note:
-ErrorAction Ignore - This ignores the exception that would arise if the storage account does not exist
Write-Host " " - Double quotes were used to allow for string interpolation since we are connecting strings and variables.
That's all.
I hope this helps