I am trying to run Azure HDInsight Cluster from PowerShell and getting error while getting storage key from my azure account :
Get-AzureSubscription -Default
$subid = (Get-AzureSubscription -Current).SubscriptionId
$clustername = "bigdatajs1"
$storagename = "bigdatajs1"
$containername = "bigdatajs1"
$creds = Get-Credential
#get storage for key
$key1 = (Get-AzureStorageKey $storagename).Primary
PS C:\Users\milind.chavan> $key1 = (Get-AzureStorageKey $storagename).Primary
Get-AzureStorageKey : ResourceNotFound: The storage account 'bigdatajs1' was not found.
By changing the subscription I can manage to get ride of this issue
Related
I have received the below error while executing my code in azureCLI task of azuredevops:
New-CosmosDbContext : The term 'New-CosmosDbContext' is not recognized as a name of a cmdlet,
what is the step to solve this or any other suggestion for insert a document in cosmosDB Container using azure CLI task in Azure Devops?
$resourceGroupName = 'rg-****'
$accountName = 'cosmosserver-****'
$databaseName = 'DefaultSystemDB'
$OrganisationName = "SampleOrg"
$ContractName = 'SampleContract'
$TypeName = 'SampleType"
$currentDate = Get-Date -Format 'yyyy-MM-dd HH:mm:ss
write-Host $currentDate
$containerName = 'Organisation'
$partitionKey = '/id'
$cosmosDbContext = New-CosmosDbContext -Account $accountName -Database $databaseName -ResourceGroup $resourceGroupName
$Organisationid='$([Guid]::NewGuid().ToString())'
write-Host Organisationid: $Organisationid
$document = #"
{
"Name": '$OrganisationName',
"CreatedOn": '$currentDate',
"id": '$Organisationid'
}
"#
$partitionkey = '$Organisationid'
New-CosmosDbDocument -Context $cosmosDbContext -CollectionId $containerName -DocumentBody $document -PartitionKey $partitionkey
Please install the CosmosDB Module first. The module is not in the hosted agent by default. Please add the following at the beginning of your script. It may need some minutes to install it.
Install-Module -Name CosmosDB -force
Edit:
In your script, you are using the New-CosmosDbContext. So you should install the CosmosDB Module instead of Az.CosmosDB Module . Please refer https://github.com/PlagueHO/CosmosDB#installation.
If you want to use the Az.CosmosDB Module , please use New-AzCosmosDBxxxxxx . See https://learn.microsoft.com/en-us/powershell/module/az.cosmosdb/?view=azps-8.1.0 for more detail.
I am trying to create HDInsight cluster in Azure with Metastore using the Powershell script. But it is throwing BadRequest: RegionCapabilityNotAvailable,Region capability not available for region 'East US' error. But East US is a supported region for the HDInsight cluster. Please find my code below.
$storageAccountResourceGroupName = "hdi-rg"
$storageAccountName = "qwertyhdi"
#$storageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $storageAccountResourceGroupName -Name $storageAccountName)[0].value
$storageContainer = "qwertyiopasdf-2020-05-03t08-30-23-118z"
# Cluster configuration info
$location = "East US"
$clusterResourceGroupName = "hdi-rg"
$clusterName = "qwertyiopasdf"
$username = "admin"
$password = ConvertTo-SecureString "password" -AsPlainText -Force
$clusterCreds = New-Object System.Management.Automation.PSCredential -ArgumentList ($username, $password)
# Hive metastore info
$hiveSqlServer = "server"
$hiveDb = "db123"
$sqlusername = "qwerty"
$sqlpassword = ConvertTo-SecureString "password" -AsPlainText -Force
$hiveCreds = New-Object System.Management.Automation.PSCredential -ArgumentList ($sqlusername, $sqlpassword)
New-AzStorageAccount `
-ResourceGroupName $storageAccountResourceGroupName `
-Name $storageAccountName `
-Location $location `
-SkuName Standard_LRS `
-Kind StorageV2 `
-EnableHttpsTrafficOnly 1
# Note: Storage account kind BlobStorage cannot be used as primary storage.
$storageAccountKey = (Get-AzStorageAccountKey `
-ResourceGroupName $storageAccountResourceGroupName `
-Name $storageAccountName)[0].Value
$defaultStorageContext = New-AzStorageContext `
-StorageAccountName $storageAccountName `
-StorageAccountKey $storageAccountKey
New-AzStorageContainer `
-Name $storageContainer `
-Context $defaultStorageContext #use the cluster name as the container name
$metastore = New-AzHDInsightClusterConfig | Add-AzHDInsightMetastore -SqlAzureServerName "$hiveSqlServer.database.windows.net" -DatabaseName $hiveDb -Credential $hiveCreds -MetastoreType HiveMetastore
New-AzHDInsightCluster -Location $location -ResourceGroupName $clusterResourceGroupName -ClusterName $clusterName -ClusterType Hadoop -OSType Windows -ClusterSizeInNodes 1 -HttpCredential $clusterCreds -DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" -DefaultStorageAccountKey $storageAccountKey -DefaultStorageContainer $storageContainer -Config $metastore
Is -OSType Windows still valid. I realize "Windows" is listed as an option from the PowerShell specs, but I think "linux" is the only value that will actually work.
You will receive this error message BadRequest: RegionCapabilityNotAvailable,Region capability not available for region 'East US', when underlying compute sku is not available for the selected region in that subscription.
I would request you to check if the resource are available from Azure Portal.
Azure Portal => Select your subscription => Usage + Quotas
Filter with => Microsoft.Compute and Region => East US
If you're unable to find a suitable SKU in that region/zone or an alternative region/zone that meets your business needs, submit a SKU request or Quota increase to Azure Support.
Running this script in Azure:
Write-Host "Running ps_example.ps1"
$resourceGroupName = 'myGroupName'
$storageName = "psexample"
$storageType = "Standard_LRS"
$location = "centralus"
if (Test-AzureName -Storage $storageName) {
Write-Host "Use existing storage account - $storageName"
} Else {
Write-Host "Make new storage account - $storageName"
New-AzureRmStorageAccount -ResourceGroupName $resourceGroupName -Name $storageName -Type $storageType -Location $location
}
The first run shows:
Running ps_example.ps1
Make new storage account - psexample
The second run shows:
Running ps_example.ps1
Make new storage account - psexample
The storage account named psexample is already taken.
Why? That would seem to indicate that if (Test-AzureName -Storage $storageName) always returns false.
If I tell Azure to use powershell 1, the version is 1.113.5. Requesting version 2.0 results in 2.0.11. The behavior is the same for both.
EDIT:
Running this:
$result = Test-AzureName -Storage $storageName
Write-Host $result
always prints False, whether psexample exists or not.
You are combining RM and SM cmdlets in Azure. Test-AzureName is a Service Management cmdlet, while New-AzureRmStorageAccount is a Resource Manager cmdlet.
You may want to try to use
if ((Get-AzureRmStorageAccountNameAvailability -Name $storageName).NameAvailable) {
Write-Host "Make new storage account - $storageName"
New-AzureRmStorageAccount -ResourceGroupName $resourceGroupName -Name $storageName -Type $storageType -Location $location
} Else {
Write-Host "Use existing storage account - $storageName"
}
to check for the name or you can create your storage account with:
New-AzureStorageAccount
Depending on what you want to use, SM or RM.
I am trying to automate an HDINSIGHT cluster using azure powershell.
I am using this template from offical doc https://github.com/Azure/azure-content/blob/master/articles/hdinsight/hdinsight-hadoop-create-linux-clusters-azure-powershell.md
How can I setup an additoinal storage account for my cluster? Do you have any idea?
Documentation mentions parameter -AdditionalStorageAccounts without examples
$resourceGroupName = "<ResourceGroupName>" # Provide the Resource Group name
$storageAccountName = "<StorageAcccountName>" # Provide the Storage account name
$containerName = "<ContainerName>" # Provide the container name
$storageAccountKey = Get-AzureStorageAccountKey -Name $storageAccountName -ResourceGroupName $resourceGroupName | %{ $_.Key1 }
# Set these variables
$clusterName = $containerName # As a best practice, have the same name for the cluster and container
$clusterNodes = <ClusterSizeInNodes> # The number of nodes in the HDInsight cluster
$credentials = Get-Credential -Message "Enter Cluster user credentials" -UserName "admin"
$sshCredentials = Get-Credential -Message "Enter SSH user credentials"
# The location of the HDInsight cluster. It must be in the same data center as the Storage account.
$location = Get-AzureRmStorageAccount -ResourceGroupName $resourceGroupName -StorageAccountName $storageAccountName | %{$_.Location}
# Create a new HDInsight cluster
New-AzureRmHDInsightCluster -ClusterName $clusterName -ResourceGroupName $resourceGroupName -HttpCredential $credentials -Location $location -DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" -DefaultStorageAccountKey $storageAccountKey -DefaultStorageContainer $containerName -ClusterSizeInNodes $clusterNodes -ClusterType Hadoop -OSType Linux -Version "3.2" -SshCredential $sshCredentials
Here is an example from the C# library, I have not used this code in about 2 year so the api might have changed, but used to work, hope it helps.
// PROVIDE THE CERTIFICATE THUMBPRINT TO RETRIEVE THE CERTIFICATE FROM THE CERTIFICATE STORE
var store = new X509Store(StoreLocation.CurrentUser);
store.Open(OpenFlags.ReadOnly);
var cert = store.Certificates.Cast<X509Certificate2>().First(item => item.Thumbprint == thumbprint);
// CREATE AN HDINSIGHT CLIENT OBJECT
var creds = new HDInsightCertificateCredential(Guid.Parse(subscriptionid), cert);
var client = HDInsightClient.Connect(creds);
client.IgnoreSslErrors = true;
// the location of additional-libs that will get pulled into the the env on create
string hiveAddtionalLibContainer = "additional-hive-lib";
var hiveAdditionalLibStorage = new WabStorageAccountConfiguration(storageaccountname, storageaccountkey, hiveAddtionalLibContainer);
// PROVIDE THE CLUSTER INFORMATION
var clusterInfo = new ClusterCreateParametersV2()
{
Name = clusterName,
Location = location,
DefaultStorageAccountName = storageaccountname,
DefaultStorageAccountKey = storageaccountkey,
DefaultStorageContainer = clusterName,
UserName = username,
Password = password,
ClusterSizeInNodes = clustersize,
Version = "3.2",
ClusterType = Microsoft.WindowsAzure.Management.HDInsight.ClusterProvisioning.Data.ClusterType.Hadoop,
};
// add more storage
clusterInfo.AdditionalStorageAccounts.Add(new WabStorageAccountConfiguration(storageaccountnameAdd1, storageaccountkeyAdd1));
client.CreateCluster(clusterInfo);
It seems we should specify the same name for cluster and default container in order to create correctly your HDInsight Cluster.
Roberto
The following lines create a storage account, and add it as an additional storage account
# create a storage account
$additionalStorageAccountName = $token + "store2"
New-AzureRmStorageAccount -ResourceGroupName $resourceGroupName -StorageAccountName $additionalStorageAccountName -Location $location -Type Standard_LRS
$additionalStorageAccountKey = (Get-AzureRmStorageAccountKey -Name $additionalStorageAccountName -ResourceGroupName $resourceGroupName)[0].Value
# Specify the additional storage account
$config = New-AzureRmHDInsightClusterConfig
Add-AzureRmHDInsightStorage -Config $config -StorageAccountName "$additionalStorageAccountName.blob.core.windows.net" -StorageAccountKey $additionalStorageAccountKey
# Create a new HDInsight cluster with an additional storage account
New-AzureRmHDInsightCluster `
-ClusterName $clusterName `
-ResourceGroupName $resourceGroupName `
-HttpCredential $credentials `
-Location $location `
-DefaultStorageAccountName "$defaultStorageAccountName.blob.core.windows.net" `
-DefaultStorageAccountKey $defaultStorageAccountKey `
-DefaultStorageContainer $defaultStorageContainerName `
-ClusterSizeInNodes $clusterNodes `
-ClusterType Hadoop `
-OSType Linux `
-Version "3.4" `
-SshCredential $sshCredentials `
-Config $config
I am building a power shell script to automate the setup of a website environment in Azure. This web uses an account storage. I want to the script not to create the account storage if exists.
I thought that using Get-AzureStorageAccount this way may work but it does not:
Write-Verbose "[Start] creating $Name storage account $Location location"
$storageAcct = Get-AzureStorageAccount –StorageAccountName $Name
if (!$storageAcct)
{
$storageAcct = New-AzureStorageAccount -StorageAccountName $Name -Location $Location -Verbose
if ($storageAcct)
{
Write-Verbose "[Finish] creating $Name storage account in $Location location"
}
else
{
throw "Failed to create a Windows Azure storage account. Failure in New-AzureStorage.ps1"
}
}
else
{
Write-Verbose "$Name storage account in $Location location already exists, skipping creation"
}
The issue is I don't know how to handle the return of Get-AzureStorageAccount.
Thank you very much in advance!
I would suggest using the Test-AzureName cmdlet to determine if it exists. So, something like this.
if (!(Test-AzureName -Storage $Name))
{
Write-Host "Creating Storage Account $Name"
New-AzureStorageAccount -StorageAccountName $Name -Location $Location
}
You can use Test-AzureName for other services too, such as Cloud Services, WebSites, and ServiceBus. It returns True if it exists, False otherwise.
Get-AzureRmStorageAccountNameAvailability -Name "accountname"
Try this:
$Name = "myStorageAccount"
$Location = "myLocation"
Write-Host "[Start] creating $Name storage account $Location location"
try{
Get-AzureStorageAccount –StorageAccountName $Name -ErrorAction Stop | Out-Null
Write-Host "$Name storage account in $Location location already exists, skipping creation"
}
catch{
Write-Host "[Finish] creating $Name storage account in $Location location"
New-AzureStorageAccount -StorageAccountName $Name -Location $Location -Verbose
}
Test-AzureName didn't work with our build agents and we already had a try/catch in code so a second one would require building it out as a function. I opted for that standard get and check if null, use -ErrorAction Ignore to stop it throwing an exception
# Check for storage account and create if not found
$StorageAccount = Get-AzureRmStorageAccount -Name $StorageAccountName -ResourceGroupName $StorageAccountRG -ErrorAction Ignore
if ($StorageAccount -eq $null)
{
New-AzureRmStorageAccount -Location "West Europe" -Name $StorageAccountName -ResourceGroupName $StorageAccountRG -SkuName Standard_LRS -Kind Storage
$StorageAccount = Get-AzureRmStorageAccount -Name $StorageAccountName -ResourceGroupName $StorageAccountRG
}
#Rick Rainey's solution works if you're logged in using Add-AzureAccount. However, Azure and powershell have a conflicting and confusing suite of login accounts (Windows Live versus AD) and login mechanisms (Classic: Add-AzureAccount; Resource manager: Login-AzureRmAccount). Some Azure powershell cmdlets require a specific login; further, some require a specific account type!
To clear through this thicket of complicated, undocumented, and confusing permission issues, we always use an AD account, logging in via Login-AzureRmAccount. We also use Azure resource manager (ARM) resources and cmdlets, following Microsoft's movement to ARM as its recommended and strategic approach. However, #RIck's solution is one which the ARM login doesn't work with. :-( So you need another approach, which is #Darren's (for storage). However, for a generic replacement for Test-AzureName I'd suggest Find-AzureRmResource. In the case of storage
$StorageObject = Find-AzureRmResource -ResourceType "Microsoft.Storage/storageAccounts" | Where-Object {$_.Name -eq $storageName}
if ( !$StorageObject ) {
$storageLocation = (Get-AzureRmResourceGroup -ResourceGroupName $resourceGroup).Location
$storageType = "Standard_LRS"
New-AzureRmStorageAccount -ResourceGroupName $resourceGroup -Name $storageName -Location $storageLocation -Type $storageType
}
You should use the latest Powershell module Az.
if ($(Get-AzStorageAccount -ResourceGroupName $resourceGroupName -Name $storageAccountName) -eq $null)
{
# does not exist
}
With the current Az module for PowerShell Version 7, the Get-AzStorageAccountNameAvailability cmdlet might offer a more efficient solution as it was designed specifically for this task. Here is an example:
# ... declare variables and specify values ...
$checkNameAvail = (Get-AzStorageAccountNameAvailability -Name $storageAccountName) | `
Select-Object NameAvailable
if ($checkNameAvail.NameAvailable)
{
Write-Host 'Account name available! Please wait while your resource is being created'
# Create account. Variables used in this example would have been declared earlier in the script.
$storageAccount = (New-AzStorageAccount -ResourceGroupName $resourceGroupName `
-AccountName $storageAccountName `
-Location $location `
-SkuName $skuType `
-AllowBlobPublicAccess $false -EnableHttpsTrafficOnly $true)
# ...
}
else
{
# This section of the script executes if the name is not available
Write-Host "The name <$storageAccountName> is not available. Suggest a new globally unique name!"
}
The condition above will return False, and execute the else statement because the boolean value returned by the cmdlet is in [0] as shown in the PowerShell command-line test below. The availability information (boolean) can thus be stripped from the object returned by the cmdlet and (as in this example) used as a condition in the rest of the script.
PS C:\> Get-AzStorageAccountNameAvailability -Name testaccount1
NameAvailable Reason Message
------------- ------ -------
False AlreadyExists The storage account named testaccount1 is already taken.
Use the error variable
Get-AzStorageAccount -ResourceGroupName 'RG-QA-TEST' -Name 'staccountfor12334ff' -ErrorVariable ev1 -ErrorAction SilentlyContinue
if ($ev1) {
Write-Host "-------------------------- Creating OEM Storage"
//create storage account
}
I had this challenge when setting up Azure storage accounts for Static website hosting using Powershell in Octopus Deploy.
Here's how I fixed it:
Using the Az module for Azure Powershell I did the following:
# Define Variables
$RESOURCE_GROUP_NAME = my-resource-group
$LOCATION = northeurope
$STORAGE_ACCOUNT_NAME = myapplication
$SKU_NAME = Standard_GRS
$STORAGE_KIND = StorageV2
# Check Storage Account and Create if not Found
$STORAGE_ACCOUNT = Get-AzStorageAccount -ResourceGroupName $RESOURCE_GROUP_NAME -Name $STORAGE_ACCOUNT_NAME -ErrorAction Ignore
if ($STORAGE_ACCOUNT -eq $null) {
Write-Host 'Creating storage account'
New-AzStorageAccount -ResourceGroupName $RESOURCE_GROUP_NAME -AccountName $STORAGE_ACCOUNT_NAME -Location $LOCATION -SkuName $SKU_NAME -Kind $STORAGE_KIND
Write-Host "$STORAGE_ACCOUNT_NAME storage account successfully created"
}
else {
Write-Host "$STORAGE_ACCOUNT_NAME storage account already exists"
}
Note:
-ErrorAction Ignore - This ignores the exception that would arise if the storage account does not exist
Write-Host " " - Double quotes were used to allow for string interpolation since we are connecting strings and variables.
That's all.
I hope this helps