I was trying to create HDInsight cluster with Hive metastore. Cluster was created successfully but without Hive metastore. Also any error was not returned. I used Powershell script below.
$resourceGroupName = "ResourceGroup"
$storageAccountName = "myStorage"
$containerName="myContainer"
$clusterName = "myCluster"
$location = "West Europe"
$clusterNodes = 2
$clusterType = "Hadoop"
$OSType = "Windows"
$hdVersion = "3.2"
$hadoopUserName = "user"
$hadoopUserPassword = "password"
$hadoopUserPW = ConvertTo-SecureString -String $hadoopUserPassword -AsPlainText -Force
$clusterCreds = New-Object System.Management.Automation.PSCredential($hadoopUserName,$hadoopUserPW)
$sqlDatabaseServerName = "mydbserver"
$metaStoreDBName = "HiveMetaStoreDB"
$sqlDatabaseLogin = "myDBLogin"
$sqlDatabasePassword = "myDBPassword"
$sqlDatabaseSecurePassword = ConvertTo-SecureString -String $sqlDatabasePassword -AsPlainText -Force
$sqlServerCred = New-Object System.Management.Automation.PSCredential ($sqlDatabaseLogin, $sqlDatabaseSecurePassword)
# Create a new HDInsight cluster
$config = New-AzureRmHDInsightClusterConfig -ClusterType Hadoop `
| Add-AzureRmHDInsightMetastore `
-SqlAzureServerName "$sqlDatabaseServerName.database.windows.net" `
-DatabaseName $metaStoreDBName `
-Credential $sqlServerCred `
-MetastoreType HiveMetastore
$config.DefaultStorageAccountName = "$storageAccountName.blob.core.windows.net"
$config.DefaultStorageAccountKey = $storageAccountKey
New-AzureRmHDInsightCluster `
-config $config `
-OSType $OSType `
-clustername $clusterName `
-HttpCredential $clusterCreds `
-DefaultStorageContainer $containerName `
-Location $location `
-ResourceGroupName $resourceGroupName `
-ClusterSizeInNodes $clusterNodes `
-Version $hdVersion
Could somebody tell me the reason why Hive metastore was not created? Or what should I do to create it?
The cluster creation process doesn't create the Azure Database for you. You must create the database before you can use it as the Hive metastore.
Related
I am trying to use PowerShell to set up an Event Hub with Capture set. However, I am experiencing problems when trying specify the capture settings with Set-AzEventHub. I have the following script commands:
$ehResourceGroup = 'kv-audit-resource'
$location = 'eastus'
$ehNameSpace = 'kv-audit-eh'
$ehName = 'security-logs'
$partitions = 1
$week = 7
# Creat resource group for hub
New-AzResourceGroup -Name $ehResourceGroup -Location $location
# Create namespace for hub
New-AzEventHubNamespace -ResourceGroupName $ehResourceGroup -NamespaceName $ehNameSpace -Location $location
# Make the hub
New-AzEventHub -ResourceGroupName $ehResourceGroup -NamespaceName $ehNameSpace -Name $ehName -PartitionCount $partitions -MessageRetentionInDays $week
# Get hub info
$loggingEventHub = Get-AzEventHub -ResourceGroupName $ehResourceGroup -NamespaceName $ehNameSpace -Name $ehName
# Add capture info
$loggingEventHub.CaptureDescription = New-Object -TypeName Microsoft.Azure.Commands.EventHub.Models.PSCaptureDescriptionAttributes
$loggingEventHub.CaptureDescription.Enabled = $true
$loggingEventHub.CaptureDescription.IntervalInSeconds = 60
$loggingEventHub.CaptureDescription.Encoding = "Avro"
$loggingEventHub.CaptureDescription.SizeLimitInBytes = 10485763
$loggingEventHub.CaptureDescription.Destination.Name = "EventHubArchive.AzureBlockBlob"
$loggingEventHub.CaptureDescription.Destination.BlobContainer = "cyberstorageaccount2"
$loggingEventHub.CaptureDescription.Destination.ArchiveNameFormat = "{Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}"
$loggingEventHub.CaptureDescription.Destination.StorageAccountResourceId = "/subscriptions/{SubscriptionId}/resourceGroups/$ehResourceGroup/providers/Microsoft.ClassicStorage/cyberstorageaccount2"
# Now update the hub with capture info
Set-AzEventHub -ResourceGroupName $ehResourceGroup -Namespace $ehNameSpace -Name $ehName -InputObject $loggingEventHub
I execute the first three commands to create the Event Hub without problems. The storage account, cyberstorageaccount2, already exists. I wait until the Azure dashboard shows that the Event Hub is successfully made and active, which takes several minutes. When I execute the last two commands to modify the capture settings, I get:
PS C:\> $loggingEventHub = Get-AzEventHub -ResourceGroupName $ehResourceGroup -NamespaceName $ehNameSpace -Name $ehName
>> $loggingEventHub.CaptureDescription = New-Object -TypeName Microsoft.Azure.Commands.EventHub.Models.PSCaptureDescriptionAttributes
>> $loggingEventHub.CaptureDescription.Enabled = $true
>> $loggingEventHub.CaptureDescription.IntervalInSeconds = 60
>> $loggingEventHub.CaptureDescription.Encoding = "Avro"
>> $loggingEventHub.CaptureDescription.SizeLimitInBytes = 10485763
>> $loggingEventHub.CaptureDescription.Destination.Name = "EventHubArchive.AzureBlockBlob"
>> $loggingEventHub.CaptureDescription.Destination.BlobContainer = "cyberstorageaccount2"
>> $loggingEventHub.CaptureDescription.Destination.ArchiveNameFormat = "{Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}"
>> $loggingEventHub.CaptureDescription.Destination.StorageAccountResourceId = "/subscriptions/{SubscriptionId}/resourceGroups/$ehResourceGroup/providers/Microsoft.ClassicStorage/cyberstorageaccount2"
>> Set-AzEventHub -ResourceGroupName $ehResourceGroup -Namespace $ehNameSpace -Name $ehName -InputObject $loggingEventHub
Set-AzEventHub : Operation returned an invalid status code 'BadRequest'
At line:11 char:1
+ Set-AzEventHub -ResourceGroupName $ehResourceGroup -Namespace $ehName ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OpenError: (Microsoft.Azure...ExecuteCmdlet():ErrorResponseException) [Set-AzEventHub],
ErrorResponseException
+ FullyQualifiedErrorId : SubCode=40000. StorageAccountResourceId. TrackingId:ea7e4590-486a-4597-9351-b5e8508857b8
_M6CH3_M6CH3_G28, SystemTracker:kv-audit-eh.servicebus.windows.net:security-logs, Timestamp:2020-12-08T12:37:56 Co
rrelationId: 4b1edeed-6585-47f0-b14e-476614404a23,Microsoft.Azure.Commands.EventHub.Commands.EventHub.SetAzureEven
tHub
What am I doing wrong? It looks like it's griping about the storage account, but when I display the contents of $loggingEventHub.CaptureDescription.Destination.StorageAccountResourceId, the path appears correct.
Thanks in advance for any info.
Storage resource-id seems malformed. It is missing 'storageaccounts' segment.
"/subscriptions/{SubscriptionId}/resourceGroups/$ehResourceGroup/providers/Microsoft.ClassicStorage/storageAccounts/cyberstorageaccount2"
I got this to work. My code was based on an example provided by Microsoft, but their example assumed you had set up a lot of things previously. What follows is my PowerShell code that fills in some of the gaps in their example. The code is commented (slightly) and has debug output that you can delete. It assumes that you have already made an Event Hub with the prerequisite resource group and namespace.
$ehResourceGroup = 'kv-audit-resource'
$location = 'eastus'
$ehNameSpace = 'kv-audit-eh'
$ehName = 'security-logs'
$partitions = 1
$week = 7
$kvStorageAccount = 'cybersecurityaccount2'
$kvContainer = 'security-container'
# Create the storage account for the Event Hub
Write-Output("Create storage account $kvStorageAccount")
$storageAcct = New-AzStorageAccount -ResourceGroupName $ehResourceGroup -AccountName $kvStorageAccount -Location $location -SkuName Standard_GRS -Kind BlobStorage -AccessTier Cool
# Get the "context" required for the container
$accountContext = $storageAcct.Context
# Create a container for the storage account
Write-Output("Create storage account container $kvContainer")
New-AzStorageContainer -Name $kvContainer -Context $accountContext -Permission 'Container'
# Get the storage account ID for the Event Hub
Write-Output("Get ID of storage account $kvStorageAccount")
$kvStorageAccountId = (Get-AzStorageAccount -ResourceGroupName $ehResourceGroup -Name $kvStorageAccount).Id
# Get Event Hub object
Write-Output("Get Event Hub object")
$loggingEventHub = Get-AzEventHub -ResourceGroupName $ehResourceGroup -Namespace $ehNameSpace -Name $ehName
# Update Event Hub capture description
Write-Output("Update Event Hub object")
$loggingEventHub.CaptureDescription = New-Object -TypeName Microsoft.Azure.Commands.EventHub.Models.PSCaptureDescriptionAttributes
$loggingEventHub.CaptureDescription.Enabled = $true
$loggingEventHub.CaptureDescription.IntervalInSeconds = 60
$loggingEventHub.CaptureDescription.Encoding = "Avro"
$loggingEventHub.CaptureDescription.SizeLimitInBytes = 10485763
$loggingEventHub.CaptureDescription.Destination.Name = "EventHubArchive.AzureBlockBlob"
$loggingEventHub.CaptureDescription.Destination.BlobContainer = $kvContainer
$loggingEventHub.CaptureDescription.Destination.ArchiveNameFormat = "{Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}"
$loggingEventHub.CaptureDescription.Destination.StorageAccountResourceId = $kvStorageAccountId
Set-AzEventHub -ResourceGroupName $ehResourceGroup -Namespace $ehNameSpace -Name $ehName -InputObject $loggingEventHub
Write-Output($loggingEventHub)
I am trying to create HDInsight cluster in Azure with Metastore using the Powershell script. But it is throwing BadRequest: RegionCapabilityNotAvailable,Region capability not available for region 'East US' error. But East US is a supported region for the HDInsight cluster. Please find my code below.
$storageAccountResourceGroupName = "hdi-rg"
$storageAccountName = "qwertyhdi"
#$storageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $storageAccountResourceGroupName -Name $storageAccountName)[0].value
$storageContainer = "qwertyiopasdf-2020-05-03t08-30-23-118z"
# Cluster configuration info
$location = "East US"
$clusterResourceGroupName = "hdi-rg"
$clusterName = "qwertyiopasdf"
$username = "admin"
$password = ConvertTo-SecureString "password" -AsPlainText -Force
$clusterCreds = New-Object System.Management.Automation.PSCredential -ArgumentList ($username, $password)
# Hive metastore info
$hiveSqlServer = "server"
$hiveDb = "db123"
$sqlusername = "qwerty"
$sqlpassword = ConvertTo-SecureString "password" -AsPlainText -Force
$hiveCreds = New-Object System.Management.Automation.PSCredential -ArgumentList ($sqlusername, $sqlpassword)
New-AzStorageAccount `
-ResourceGroupName $storageAccountResourceGroupName `
-Name $storageAccountName `
-Location $location `
-SkuName Standard_LRS `
-Kind StorageV2 `
-EnableHttpsTrafficOnly 1
# Note: Storage account kind BlobStorage cannot be used as primary storage.
$storageAccountKey = (Get-AzStorageAccountKey `
-ResourceGroupName $storageAccountResourceGroupName `
-Name $storageAccountName)[0].Value
$defaultStorageContext = New-AzStorageContext `
-StorageAccountName $storageAccountName `
-StorageAccountKey $storageAccountKey
New-AzStorageContainer `
-Name $storageContainer `
-Context $defaultStorageContext #use the cluster name as the container name
$metastore = New-AzHDInsightClusterConfig | Add-AzHDInsightMetastore -SqlAzureServerName "$hiveSqlServer.database.windows.net" -DatabaseName $hiveDb -Credential $hiveCreds -MetastoreType HiveMetastore
New-AzHDInsightCluster -Location $location -ResourceGroupName $clusterResourceGroupName -ClusterName $clusterName -ClusterType Hadoop -OSType Windows -ClusterSizeInNodes 1 -HttpCredential $clusterCreds -DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" -DefaultStorageAccountKey $storageAccountKey -DefaultStorageContainer $storageContainer -Config $metastore
Is -OSType Windows still valid. I realize "Windows" is listed as an option from the PowerShell specs, but I think "linux" is the only value that will actually work.
You will receive this error message BadRequest: RegionCapabilityNotAvailable,Region capability not available for region 'East US', when underlying compute sku is not available for the selected region in that subscription.
I would request you to check if the resource are available from Azure Portal.
Azure Portal => Select your subscription => Usage + Quotas
Filter with => Microsoft.Compute and Region => East US
If you're unable to find a suitable SKU in that region/zone or an alternative region/zone that meets your business needs, submit a SKU request or Quota increase to Azure Support.
I am doing batch uploads from a csv file to Azure table storage through a Powershell script and i have a command: $table.CloudTable.ExecuteBatch($batchOperation)
for which i'm getting the error mentioned in the header of the question of my post. I believe that "ExecuteBatch" is a method in the old AzureRm module and not the newer Az module which i am using, which is causing it to break. Is there a corresponding method in Az module for "ExecuteBatch"?
According to my test, if we use new Azure PowerShell module Az to manage Azure Table storage, we need to use the SDK Microsoft.Azure.Cosmos.Table.
So if ou want to use ExecuteBatch method, we need to use the command [Microsoft.Azure.Cosmos.Table.TableBatchOperation] $batchOperation = New-Object -TypeName Microsoft.Azure.Cosmos.Table.TableBatchOperation to create TableBatchOperation. For example:
Connect-AzAccount
$ResourceGroupName = "testfun06"
$StorageAccountName="testfun06bf01"
$TableName="People"
$keys=Get-AzStorageAccountKey -ResourceGroupName $ResourceGroupName -Name $StorageAccountName
$ctx = New-AzStorageContext -StorageAccountName $StorageAccountName -StorageAccountKey $keys[0].Value
$table = Get-AzStorageTable -Name $TableName -Context $ctx
$e = New-Object Microsoft.Azure.Cosmos.Table.DynamicTableEntity("Jim","test")
$e1 = New-Object Microsoft.Azure.Cosmos.Table.DynamicTableEntity("Jim","test1")
[Microsoft.Azure.Cosmos.Table.TableBatchOperation] $batchOperation = New-Object -TypeName Microsoft.Azure.Cosmos.Table.TableBatchOperation
$batchOperation.InsertOrMerge($e)
$batchOperation.InsertOrMerge($e1)
$table.CloudTable.ExecuteBatch($batchOperation)
Looking for a script to autogenerate a new storage account
with a prefix dev something like dev01.... and when i rerun the template should increment as dev02.. on second run and so on.
I tried giving the parameters / using default templates in github.
Issue is if i pass on a value under the value the system deploys it fine ,if i give the same name it would rerun and update the existing storage.
Instead i would like it to check if storage account exists and if not create a new account , please advise any pointers if any
Sample Parameters.Json file im using :
"parameters": {
"storageAccountName": {
"value": "dev01"
},
I recommand that you could use the Azure powershell script to customize your logic to do that.
The following is the demo code:
$resourceGroup = "rgName"
$storageAccount = "accountName"
$location = "Central US"
$SkuName = "Standard_LRS"
$kind = "StorageV2"
$i = 0;
while(1)
{
$storage = Get-AzureRmStorageAccount -ResourceGroupName $resourceGroup -Name $storageAccount
if($storage -ne $null)
{
$i++
$storageAccount = $storage.StorageAccountName + $i
$storage = New-AzureRmStorageAccount -ResourceGroupName $resourceGroup -Name $storageAccount -SkuName $SkuName -Location $location -Kind $kind
}
else
{
$storage = New-AzureRmStorageAccount -ResourceGroupName $resourceGroup -Name $storageAccount -SkuName $SkuName -Location $location -Kind $kind
$storageAccount = $storageAccount +$i;
}
if ($storage -ne $null)
{
break;
}
}
Task:
I am trying to automate an HDINSIGHT cluster using azure powershell.
I am using this template from offical doc https://github.com/Azure/azure-content/blob/master/articles/hdinsight/hdinsight-hadoop-create-linux-clusters-azure-powershell.md
How can I setup an additoinal storage account for my cluster? Do you have any idea?
Documentation mentions parameter -AdditionalStorageAccounts without examples
$resourceGroupName = "<ResourceGroupName>" # Provide the Resource Group name
$storageAccountName = "<StorageAcccountName>" # Provide the Storage account name
$containerName = "<ContainerName>" # Provide the container name
$storageAccountKey = Get-AzureStorageAccountKey -Name $storageAccountName -ResourceGroupName $resourceGroupName | %{ $_.Key1 }
# Set these variables
$clusterName = $containerName # As a best practice, have the same name for the cluster and container
$clusterNodes = <ClusterSizeInNodes> # The number of nodes in the HDInsight cluster
$credentials = Get-Credential -Message "Enter Cluster user credentials" -UserName "admin"
$sshCredentials = Get-Credential -Message "Enter SSH user credentials"
# The location of the HDInsight cluster. It must be in the same data center as the Storage account.
$location = Get-AzureRmStorageAccount -ResourceGroupName $resourceGroupName -StorageAccountName $storageAccountName | %{$_.Location}
# Create a new HDInsight cluster
New-AzureRmHDInsightCluster -ClusterName $clusterName -ResourceGroupName $resourceGroupName -HttpCredential $credentials -Location $location -DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" -DefaultStorageAccountKey $storageAccountKey -DefaultStorageContainer $containerName -ClusterSizeInNodes $clusterNodes -ClusterType Hadoop -OSType Linux -Version "3.2" -SshCredential $sshCredentials
Here is an example from the C# library, I have not used this code in about 2 year so the api might have changed, but used to work, hope it helps.
// PROVIDE THE CERTIFICATE THUMBPRINT TO RETRIEVE THE CERTIFICATE FROM THE CERTIFICATE STORE
var store = new X509Store(StoreLocation.CurrentUser);
store.Open(OpenFlags.ReadOnly);
var cert = store.Certificates.Cast<X509Certificate2>().First(item => item.Thumbprint == thumbprint);
// CREATE AN HDINSIGHT CLIENT OBJECT
var creds = new HDInsightCertificateCredential(Guid.Parse(subscriptionid), cert);
var client = HDInsightClient.Connect(creds);
client.IgnoreSslErrors = true;
// the location of additional-libs that will get pulled into the the env on create
string hiveAddtionalLibContainer = "additional-hive-lib";
var hiveAdditionalLibStorage = new WabStorageAccountConfiguration(storageaccountname, storageaccountkey, hiveAddtionalLibContainer);
// PROVIDE THE CLUSTER INFORMATION
var clusterInfo = new ClusterCreateParametersV2()
{
Name = clusterName,
Location = location,
DefaultStorageAccountName = storageaccountname,
DefaultStorageAccountKey = storageaccountkey,
DefaultStorageContainer = clusterName,
UserName = username,
Password = password,
ClusterSizeInNodes = clustersize,
Version = "3.2",
ClusterType = Microsoft.WindowsAzure.Management.HDInsight.ClusterProvisioning.Data.ClusterType.Hadoop,
};
// add more storage
clusterInfo.AdditionalStorageAccounts.Add(new WabStorageAccountConfiguration(storageaccountnameAdd1, storageaccountkeyAdd1));
client.CreateCluster(clusterInfo);
It seems we should specify the same name for cluster and default container in order to create correctly your HDInsight Cluster.
Roberto
The following lines create a storage account, and add it as an additional storage account
# create a storage account
$additionalStorageAccountName = $token + "store2"
New-AzureRmStorageAccount -ResourceGroupName $resourceGroupName -StorageAccountName $additionalStorageAccountName -Location $location -Type Standard_LRS
$additionalStorageAccountKey = (Get-AzureRmStorageAccountKey -Name $additionalStorageAccountName -ResourceGroupName $resourceGroupName)[0].Value
# Specify the additional storage account
$config = New-AzureRmHDInsightClusterConfig
Add-AzureRmHDInsightStorage -Config $config -StorageAccountName "$additionalStorageAccountName.blob.core.windows.net" -StorageAccountKey $additionalStorageAccountKey
# Create a new HDInsight cluster with an additional storage account
New-AzureRmHDInsightCluster `
-ClusterName $clusterName `
-ResourceGroupName $resourceGroupName `
-HttpCredential $credentials `
-Location $location `
-DefaultStorageAccountName "$defaultStorageAccountName.blob.core.windows.net" `
-DefaultStorageAccountKey $defaultStorageAccountKey `
-DefaultStorageContainer $defaultStorageContainerName `
-ClusterSizeInNodes $clusterNodes `
-ClusterType Hadoop `
-OSType Linux `
-Version "3.4" `
-SshCredential $sshCredentials `
-Config $config