I am trying to create HDInsight cluster in Azure with Metastore using the Powershell script. But it is throwing BadRequest: RegionCapabilityNotAvailable,Region capability not available for region 'East US' error. But East US is a supported region for the HDInsight cluster. Please find my code below.
$storageAccountResourceGroupName = "hdi-rg"
$storageAccountName = "qwertyhdi"
#$storageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $storageAccountResourceGroupName -Name $storageAccountName)[0].value
$storageContainer = "qwertyiopasdf-2020-05-03t08-30-23-118z"
# Cluster configuration info
$location = "East US"
$clusterResourceGroupName = "hdi-rg"
$clusterName = "qwertyiopasdf"
$username = "admin"
$password = ConvertTo-SecureString "password" -AsPlainText -Force
$clusterCreds = New-Object System.Management.Automation.PSCredential -ArgumentList ($username, $password)
# Hive metastore info
$hiveSqlServer = "server"
$hiveDb = "db123"
$sqlusername = "qwerty"
$sqlpassword = ConvertTo-SecureString "password" -AsPlainText -Force
$hiveCreds = New-Object System.Management.Automation.PSCredential -ArgumentList ($sqlusername, $sqlpassword)
New-AzStorageAccount `
-ResourceGroupName $storageAccountResourceGroupName `
-Name $storageAccountName `
-Location $location `
-SkuName Standard_LRS `
-Kind StorageV2 `
-EnableHttpsTrafficOnly 1
# Note: Storage account kind BlobStorage cannot be used as primary storage.
$storageAccountKey = (Get-AzStorageAccountKey `
-ResourceGroupName $storageAccountResourceGroupName `
-Name $storageAccountName)[0].Value
$defaultStorageContext = New-AzStorageContext `
-StorageAccountName $storageAccountName `
-StorageAccountKey $storageAccountKey
New-AzStorageContainer `
-Name $storageContainer `
-Context $defaultStorageContext #use the cluster name as the container name
$metastore = New-AzHDInsightClusterConfig | Add-AzHDInsightMetastore -SqlAzureServerName "$hiveSqlServer.database.windows.net" -DatabaseName $hiveDb -Credential $hiveCreds -MetastoreType HiveMetastore
New-AzHDInsightCluster -Location $location -ResourceGroupName $clusterResourceGroupName -ClusterName $clusterName -ClusterType Hadoop -OSType Windows -ClusterSizeInNodes 1 -HttpCredential $clusterCreds -DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" -DefaultStorageAccountKey $storageAccountKey -DefaultStorageContainer $storageContainer -Config $metastore
Is -OSType Windows still valid. I realize "Windows" is listed as an option from the PowerShell specs, but I think "linux" is the only value that will actually work.
You will receive this error message BadRequest: RegionCapabilityNotAvailable,Region capability not available for region 'East US', when underlying compute sku is not available for the selected region in that subscription.
I would request you to check if the resource are available from Azure Portal.
Azure Portal => Select your subscription => Usage + Quotas
Filter with => Microsoft.Compute and Region => East US
If you're unable to find a suitable SKU in that region/zone or an alternative region/zone that meets your business needs, submit a SKU request or Quota increase to Azure Support.
Related
I'm trying to create a service fabric cluster with a powershell script. Here's the script:
Param(
[Parameter(HelpMessage="Azure user name")]
[String]$azUserName,
[Parameter(HelpMessage="Azure password")]
[String]$azPassword,
[Parameter(HelpMessage="What is the resource group name?")]
[String]$ResourceGroupName="sfLinuxClusterRg",
[Parameter(HelpMessage="What is the key vault group name?")]
[String]$KeyVaultResourceGroupName="KeyVaultRg",
[Parameter(HelpMessage="Where is the azure location?")]
[String]$Location="uksouth",
[Parameter(HelpMessage="what is the key vault group?")]
[String]$VaultGroupName="linuxclusterkeyvaultgroup",
[Parameter(HelpMessage="Where is the certificate stored locally?")]
[String]$CertFileName="certificate.pfx",
[Parameter(HelpMessage="What is the subscription id?")]
[String]$SubscriptionId,
[Parameter(HelpMessage="What is the tenant id?")]
[String]$TenantId
)
Function DecryptSecureString([SecureString] $secureText){
return [Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($secureText))
}
Function CreateKeyVaultCert(){
try{
Select-AzureRmSubscription -SubscriptionId $SubscriptionId `
-TenantId $TenantId
# Create a Resource Group to hold your Key Vault(s)
# Note this should be separate from your other resources so you can delete those other resource groups without impacting your registered certs
New-AzureRmResourceGroup -Name $KeyVaultResourceGroupName -Location uksouth -Force
# Create a Key Vault to hold your secrets
New-AzureRmKeyVault -VaultName $vaultName -ResourceGroupName $KeyVaultResourceGroupName `
-Location uksouth -EnabledForDeployment `
-EnabledForDiskEncryption -EnabledForTemplateDeployment
# Have Key Vault create the certificate with a simple policy
$policy = New-AzureKeyVaultCertificatePolicy `
-SubjectName "CN=cluster123.uksouth.cloudapp.azure.com" `
-IssuerName Self -ValidityInMonths 12
Add-AzureKeyVaultCertificate -VaultName $vaultName -Name cert1 -CertificatePolicy $policy
Set-AzureRmKeyVaultAccessPolicy -VaultName $vaultName -EnabledForDeployment
Set-AzureRmKeyVaultAccessPolicy -VaultName $vaultName -EmailAddress $azUserName -PermissionsToSecrets Get
}
catch{
Write-Host $_.Exception.Message
exit
}
}
$azPassword = $azPassword | ConvertTo-SecureString -AsPlainText -Force
Import-AzureRmContext -Path "C:\stuff\AzureProfile.json"
$Date = Get-Date
$VaultName="VaultLearn" + $Date.ToString("hhmmss")
# sign in to your Azure account and select your subscription
$certName = "CN=cluster123.uksouth.cloudapp.azure.com"
$clustersize=5
$clustername = "cluster123" + $Date.ToString("hhmmss")
$adminuser="admin"
$vmsku = "Standard_D2_v2"
$os = "UbuntuServer1604"
Select-AzureRmSubscription -SubscriptionId $SubscriptionId
CreateKeyVaultCert
New-AzureRmServiceFabricCluster -Name $clustername `
-ResourceGroupName $ResourceGroupName `
-Location uksouth `
-ClusterSize $clustersize `
-VmUserName $adminuser -VmPassword $azPassword `
-CertificateSubjectName $certName `
-CertificatePassword $azPassword `
-VmSku $vmsku `
-CertificateOutputFolder 'C:\stuff' `
-OS $os `
-KeyVaultName $VaultName
Write-Host "Created service fabric cluster in resource group $ResourceGroupName"
#Add-AzureRmAccount
# $path = "C:\stuff\AzureProfile.json"
# Save-AzureRmContext -Path $path -Force
I keep getting the error:
New-AzureRmServiceFabricCluster : The name 'VaultLearn090903' is already in use.
This is really weird, because I'm pretty sure yesterday this script at least attempted to create the cluster, although I was getting another error about Ubuntu image not being found. What could be causing this error?
By the way, in order to run this script, I have previously run this code:
Add-AzureRmAccount
$path = "C:\stuff\AzureProfile.json"
Save-AzureRmContext -Path $path -Force
which has cached my azure credentials locally.
I have specifically added two statements into the script to add a policy for access to the keyvault:
Set-AzureRmKeyVaultAccessPolicy -VaultName $vaultName -EnabledForDeployment
Set-AzureRmKeyVaultAccessPolicy -VaultName $vaultName -EmailAddress $azUserName -PermissionsToSecrets Get
I know this is probably overkill, but I'm trying to get rid of this problem.
The other weird thing is I have tried the Azure Sample Github repo
I'm at a loss for how I can continue with service fabric.
For completeness:
I managed to get something different working by going to the azure portal, creating a service fabric cluster and downloading the ARM template (which includes some powershell scripts).
thanks
I want to upscale and downscale my Azure Analysis Services with PowerShell (Automation Runbook), but changing the Tier (Sku) doesn't seem to work. However there are no errors. Any suggestions?
# PowerShell code
# Connect to a connection to get TenantId and SubscriptionId
$Connection = Get-AutomationConnection -Name "AzureRunAsConnection"
$TenantId = $Connection.TenantId
$SubscriptionId = $Connection.SubscriptionId
# Get the service principal credentials connected to the automation account.
$null = $SPCredential = Get-AutomationPSCredential -Name "SSISJoost"
# Login to Azure ($null is to prevent output, since Out-Null doesn't work in Azure)
Write-Output "Login to Azure using automation account 'SSISJoost'."
$null = Login-AzureRmAccount -TenantId $TenantId -SubscriptionId $SubscriptionId -Credential $SPCredential
# Select the correct subscription
Write-Output "Selecting subscription '$($SubscriptionId)'."
$null = Select-AzureRmSubscription -SubscriptionID $SubscriptionId
# Get variable values
$ResourceGroupName = Get-AutomationVariable -Name 'ResourceGroupName'
$AnalysisServerName = Get-AutomationVariable -Name 'AnalysisServerName'
# Get old status (for testing/logging purpose only)
$OldAsSetting = Get-AzureRmAnalysisServicesServer -ResourceGroupName $ResourceGroupName -Name $AnalysisServerName
try
{
# changing tier
Write-Output "Upgrade $($AnalysisServerName) to S1. Current tier: $($OldAsSetting.Sku.Name)"
Set-AzureRmAnalysisServicesServer -ResourceGroupName $ResourceGroupName -Name $AnalysisServerName -Sku "S1"
}
catch
{
Write-Error -Message $_.Exception
throw $_.Exception
}
Write-Output "Done"
# Get new status (for testing/logging purpose only)
$NewAsSetting = Get-AzureRmAnalysisServicesServer -ResourceGroupName $ResourceGroupName -Name $AnalysisServerName
Write-Output "New tier: $($NewAsSetting.Sku.Name)"
using Set-AzureRmAnalysisServicesServer
There was a little bug in the PowerShell AzureRM.AnalysisServices module. It has been fixed in 0.4.0 (Thursday, June 08 2017)
Now the code finally works: http://microsoft-bitools.blogspot.com/2017/06/schedule-upscaledownscale-azure.html
I am generating SAS token from PowerShell but when I am trying to access that token from Azure Storage explorer, it is giving problem "Authentication Error. Signature fields not well formed."
here is the full Powershell command :-
Parameter required
$StorageAccountName = 'XXXXXX'
$ResourceGroup = 'remoteaccess'
$ContainerName = "vhds"
Powershell Cmd
$AzStrAct = Get-AzureRmStorageAccount -Name $StorageAccountName -ResourceGroupName $ResourceGroup
$AzStrKey = Get-AzureRmStorageAccountKey -Name $StorageAccountName -ResourceGroupName $ResourceGroup
$AzStrCtx = New-AzureStorageContext $StorageAccountName -StorageAccountKey $AzStrKey[0].Value
Get-AzureStorageContainer -Name $ContainerName -Context $AzStrCtx
$ContainerSASTokenURI = New-AzureStorageContainerSASToken -Name $ContainerName -Permission "rwdl" -StartTime "2017-04-12" -ExpiryTime "2017-04-16" -Context $AzStrCtx -FullUri
Write-Host "The SAS Token of container as below:"
$ContainerSASTokenURI
output
https://XXXXXXX.blob.core.windows.net/vhds?sv=2015-04-05&sr=c&sig=XXXXXXXXXXXXXXXXXXXXXXXX&st=2017-04-1
1T18%3A30%3A00Z&se=2017-04-15T18%3A30%3A00Z&sp=rwdl
I test with your script, and it works for me, my Azure storage explorer version is 0.8.12, I suggest you upgrade Azure storage explorer's version to 0.8.12.
I am trying to automate an HDINSIGHT cluster using azure powershell.
I am using this template from offical doc https://github.com/Azure/azure-content/blob/master/articles/hdinsight/hdinsight-hadoop-create-linux-clusters-azure-powershell.md
How can I setup an additoinal storage account for my cluster? Do you have any idea?
Documentation mentions parameter -AdditionalStorageAccounts without examples
$resourceGroupName = "<ResourceGroupName>" # Provide the Resource Group name
$storageAccountName = "<StorageAcccountName>" # Provide the Storage account name
$containerName = "<ContainerName>" # Provide the container name
$storageAccountKey = Get-AzureStorageAccountKey -Name $storageAccountName -ResourceGroupName $resourceGroupName | %{ $_.Key1 }
# Set these variables
$clusterName = $containerName # As a best practice, have the same name for the cluster and container
$clusterNodes = <ClusterSizeInNodes> # The number of nodes in the HDInsight cluster
$credentials = Get-Credential -Message "Enter Cluster user credentials" -UserName "admin"
$sshCredentials = Get-Credential -Message "Enter SSH user credentials"
# The location of the HDInsight cluster. It must be in the same data center as the Storage account.
$location = Get-AzureRmStorageAccount -ResourceGroupName $resourceGroupName -StorageAccountName $storageAccountName | %{$_.Location}
# Create a new HDInsight cluster
New-AzureRmHDInsightCluster -ClusterName $clusterName -ResourceGroupName $resourceGroupName -HttpCredential $credentials -Location $location -DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" -DefaultStorageAccountKey $storageAccountKey -DefaultStorageContainer $containerName -ClusterSizeInNodes $clusterNodes -ClusterType Hadoop -OSType Linux -Version "3.2" -SshCredential $sshCredentials
Here is an example from the C# library, I have not used this code in about 2 year so the api might have changed, but used to work, hope it helps.
// PROVIDE THE CERTIFICATE THUMBPRINT TO RETRIEVE THE CERTIFICATE FROM THE CERTIFICATE STORE
var store = new X509Store(StoreLocation.CurrentUser);
store.Open(OpenFlags.ReadOnly);
var cert = store.Certificates.Cast<X509Certificate2>().First(item => item.Thumbprint == thumbprint);
// CREATE AN HDINSIGHT CLIENT OBJECT
var creds = new HDInsightCertificateCredential(Guid.Parse(subscriptionid), cert);
var client = HDInsightClient.Connect(creds);
client.IgnoreSslErrors = true;
// the location of additional-libs that will get pulled into the the env on create
string hiveAddtionalLibContainer = "additional-hive-lib";
var hiveAdditionalLibStorage = new WabStorageAccountConfiguration(storageaccountname, storageaccountkey, hiveAddtionalLibContainer);
// PROVIDE THE CLUSTER INFORMATION
var clusterInfo = new ClusterCreateParametersV2()
{
Name = clusterName,
Location = location,
DefaultStorageAccountName = storageaccountname,
DefaultStorageAccountKey = storageaccountkey,
DefaultStorageContainer = clusterName,
UserName = username,
Password = password,
ClusterSizeInNodes = clustersize,
Version = "3.2",
ClusterType = Microsoft.WindowsAzure.Management.HDInsight.ClusterProvisioning.Data.ClusterType.Hadoop,
};
// add more storage
clusterInfo.AdditionalStorageAccounts.Add(new WabStorageAccountConfiguration(storageaccountnameAdd1, storageaccountkeyAdd1));
client.CreateCluster(clusterInfo);
It seems we should specify the same name for cluster and default container in order to create correctly your HDInsight Cluster.
Roberto
The following lines create a storage account, and add it as an additional storage account
# create a storage account
$additionalStorageAccountName = $token + "store2"
New-AzureRmStorageAccount -ResourceGroupName $resourceGroupName -StorageAccountName $additionalStorageAccountName -Location $location -Type Standard_LRS
$additionalStorageAccountKey = (Get-AzureRmStorageAccountKey -Name $additionalStorageAccountName -ResourceGroupName $resourceGroupName)[0].Value
# Specify the additional storage account
$config = New-AzureRmHDInsightClusterConfig
Add-AzureRmHDInsightStorage -Config $config -StorageAccountName "$additionalStorageAccountName.blob.core.windows.net" -StorageAccountKey $additionalStorageAccountKey
# Create a new HDInsight cluster with an additional storage account
New-AzureRmHDInsightCluster `
-ClusterName $clusterName `
-ResourceGroupName $resourceGroupName `
-HttpCredential $credentials `
-Location $location `
-DefaultStorageAccountName "$defaultStorageAccountName.blob.core.windows.net" `
-DefaultStorageAccountKey $defaultStorageAccountKey `
-DefaultStorageContainer $defaultStorageContainerName `
-ClusterSizeInNodes $clusterNodes `
-ClusterType Hadoop `
-OSType Linux `
-Version "3.4" `
-SshCredential $sshCredentials `
-Config $config
I was trying to create HDInsight cluster with Hive metastore. Cluster was created successfully but without Hive metastore. Also any error was not returned. I used Powershell script below.
$resourceGroupName = "ResourceGroup"
$storageAccountName = "myStorage"
$containerName="myContainer"
$clusterName = "myCluster"
$location = "West Europe"
$clusterNodes = 2
$clusterType = "Hadoop"
$OSType = "Windows"
$hdVersion = "3.2"
$hadoopUserName = "user"
$hadoopUserPassword = "password"
$hadoopUserPW = ConvertTo-SecureString -String $hadoopUserPassword -AsPlainText -Force
$clusterCreds = New-Object System.Management.Automation.PSCredential($hadoopUserName,$hadoopUserPW)
$sqlDatabaseServerName = "mydbserver"
$metaStoreDBName = "HiveMetaStoreDB"
$sqlDatabaseLogin = "myDBLogin"
$sqlDatabasePassword = "myDBPassword"
$sqlDatabaseSecurePassword = ConvertTo-SecureString -String $sqlDatabasePassword -AsPlainText -Force
$sqlServerCred = New-Object System.Management.Automation.PSCredential ($sqlDatabaseLogin, $sqlDatabaseSecurePassword)
# Create a new HDInsight cluster
$config = New-AzureRmHDInsightClusterConfig -ClusterType Hadoop `
| Add-AzureRmHDInsightMetastore `
-SqlAzureServerName "$sqlDatabaseServerName.database.windows.net" `
-DatabaseName $metaStoreDBName `
-Credential $sqlServerCred `
-MetastoreType HiveMetastore
$config.DefaultStorageAccountName = "$storageAccountName.blob.core.windows.net"
$config.DefaultStorageAccountKey = $storageAccountKey
New-AzureRmHDInsightCluster `
-config $config `
-OSType $OSType `
-clustername $clusterName `
-HttpCredential $clusterCreds `
-DefaultStorageContainer $containerName `
-Location $location `
-ResourceGroupName $resourceGroupName `
-ClusterSizeInNodes $clusterNodes `
-Version $hdVersion
Could somebody tell me the reason why Hive metastore was not created? Or what should I do to create it?
The cluster creation process doesn't create the Azure Database for you. You must create the database before you can use it as the Hive metastore.