I am trying to create HDInsight cluster in Azure with Metastore using the Powershell script. But it is throwing BadRequest: RegionCapabilityNotAvailable,Region capability not available for region 'East US' error. But East US is a supported region for the HDInsight cluster. Please find my code below.
$storageAccountResourceGroupName = "hdi-rg"
$storageAccountName = "qwertyhdi"
#$storageAccountKey = (Get-AzStorageAccountKey -ResourceGroupName $storageAccountResourceGroupName -Name $storageAccountName)[0].value
$storageContainer = "qwertyiopasdf-2020-05-03t08-30-23-118z"
# Cluster configuration info
$location = "East US"
$clusterResourceGroupName = "hdi-rg"
$clusterName = "qwertyiopasdf"
$username = "admin"
$password = ConvertTo-SecureString "password" -AsPlainText -Force
$clusterCreds = New-Object System.Management.Automation.PSCredential -ArgumentList ($username, $password)
# Hive metastore info
$hiveSqlServer = "server"
$hiveDb = "db123"
$sqlusername = "qwerty"
$sqlpassword = ConvertTo-SecureString "password" -AsPlainText -Force
$hiveCreds = New-Object System.Management.Automation.PSCredential -ArgumentList ($sqlusername, $sqlpassword)
New-AzStorageAccount `
-ResourceGroupName $storageAccountResourceGroupName `
-Name $storageAccountName `
-Location $location `
-SkuName Standard_LRS `
-Kind StorageV2 `
-EnableHttpsTrafficOnly 1
# Note: Storage account kind BlobStorage cannot be used as primary storage.
$storageAccountKey = (Get-AzStorageAccountKey `
-ResourceGroupName $storageAccountResourceGroupName `
-Name $storageAccountName)[0].Value
$defaultStorageContext = New-AzStorageContext `
-StorageAccountName $storageAccountName `
-StorageAccountKey $storageAccountKey
New-AzStorageContainer `
-Name $storageContainer `
-Context $defaultStorageContext #use the cluster name as the container name
$metastore = New-AzHDInsightClusterConfig | Add-AzHDInsightMetastore -SqlAzureServerName "$hiveSqlServer.database.windows.net" -DatabaseName $hiveDb -Credential $hiveCreds -MetastoreType HiveMetastore
New-AzHDInsightCluster -Location $location -ResourceGroupName $clusterResourceGroupName -ClusterName $clusterName -ClusterType Hadoop -OSType Windows -ClusterSizeInNodes 1 -HttpCredential $clusterCreds -DefaultStorageAccountName "$storageAccountName.blob.core.windows.net" -DefaultStorageAccountKey $storageAccountKey -DefaultStorageContainer $storageContainer -Config $metastore
Is -OSType Windows still valid. I realize "Windows" is listed as an option from the PowerShell specs, but I think "linux" is the only value that will actually work.
You will receive this error message BadRequest: RegionCapabilityNotAvailable,Region capability not available for region 'East US', when underlying compute sku is not available for the selected region in that subscription.
I would request you to check if the resource are available from Azure Portal.
Azure Portal => Select your subscription => Usage + Quotas
Filter with => Microsoft.Compute and Region => East US
If you're unable to find a suitable SKU in that region/zone or an alternative region/zone that meets your business needs, submit a SKU request or Quota increase to Azure Support.
Looking for a script to autogenerate a new storage account
with a prefix dev something like dev01.... and when i rerun the template should increment as dev02.. on second run and so on.
I tried giving the parameters / using default templates in github.
Issue is if i pass on a value under the value the system deploys it fine ,if i give the same name it would rerun and update the existing storage.
Instead i would like it to check if storage account exists and if not create a new account , please advise any pointers if any
Sample Parameters.Json file im using :
"parameters": {
"storageAccountName": {
"value": "dev01"
},
I recommand that you could use the Azure powershell script to customize your logic to do that.
The following is the demo code:
$resourceGroup = "rgName"
$storageAccount = "accountName"
$location = "Central US"
$SkuName = "Standard_LRS"
$kind = "StorageV2"
$i = 0;
while(1)
{
$storage = Get-AzureRmStorageAccount -ResourceGroupName $resourceGroup -Name $storageAccount
if($storage -ne $null)
{
$i++
$storageAccount = $storage.StorageAccountName + $i
$storage = New-AzureRmStorageAccount -ResourceGroupName $resourceGroup -Name $storageAccount -SkuName $SkuName -Location $location -Kind $kind
}
else
{
$storage = New-AzureRmStorageAccount -ResourceGroupName $resourceGroup -Name $storageAccount -SkuName $SkuName -Location $location -Kind $kind
$storageAccount = $storageAccount +$i;
}
if ($storage -ne $null)
{
break;
}
}
Task:
What username and password should we use to access SF nodes ?
Is it the same Azure portal username and password?
You have provided the password as part of the creation of the cluster. For example, it's passed as an ARM template parameter. Fortunately, you can reset the password of your VM scaleset.
Login-AzureRmAccount
Get-AzureRmSubscription -SubscriptionId 'SUBSCRIPTIONID' | Select-AzureRmSubscription
$vmssName = 'SCALESETNAME'
$vmssResourceGroup = 'SCALESETRESOURCEGROUP'
$publicConfig = #{'UserName' = 'USERNAME'}
$privateConfig = #{'Password' = 'NEWPASSWORD'}
$extName = 'VMAccessAgent'
$publisher = 'Microsoft.Compute'
$vmss = Get-AzureRmVmss -ResourceGroupName $vmssResourceGroup -VMScaleSetName $vmssName
$vmss = Add-AzureRmVmssExtension -VirtualMachineScaleSet $vmss -Name $extName -Publisher $publisher -Setting $publicConfig -ProtectedSetting $privateConfig -Type $extName -TypeHandlerVersion '2.0' -AutoUpgradeMinorVersion $true
Update-AzureRmVmss -ResourceGroupName $vmssResourceGroup -Name $vmssName -VirtualMachineScaleSet $vmss
I have hunted around for an answer to this, but I am not having much luck. All the articles I can find are either setting up a Point-to-Site or are instructions for classic Azure, not Azure 2.0 (Resource Group)
Currently, we are dialing up a whole new resource group everytime we do a new built. This consists of Web apps and SQL DBs. When we have a new build we start up the new and del the old resource group. Simple. To minimize the start-up time we have a static resource group that isn't deleted that houses the VPN connection to our on Prem resources.
The problem I'm having is when I add the new websites using AzureRM Powershell cmd's to the Point-to-site it says it's successful. The Azure Portal says its good but it does let me communicate. If I remove and add it from one of the 8 WebApps they all start working.
I am out of ideas. Any help would be greatly appreciated.
Azure VPN
Below is the function I have put togeather from what I can find out there.
function AddExistingVnet{
param(
[string] $subscriptionId,
[string] $resourceGroupName,
[string] $webAppName
)
$Vnet = Get-AzureRmVirtualNetwork | Where-Object {$_.ResourceGroupName -like "*Static*"}
IF($Vnet.Name.count -gt 1) {write-host 'Two or networks have been returned. Unable to continue ' return}
$gatewaySubnet = $vnet.Subnets | Where-Object { $_.Name -eq "GatewaySubnet" }
$vnetName = $vnet.Name
$uriParts = $gatewaySubnet.IpConfigurations[0].Id.Split('/')
$gatewayResourceGroup = $uriParts[4]
$gatewayName = $uriParts[8]
$gateway = Get-AzureRmVirtualNetworkGateway -ResourceGroupName $vnet.ResourceGroupName -Name $gatewayName
Write-Host "Creating App association to VNET"
$propertiesObject = #{
"vnetResourceId" = "/subscriptions/$($subscriptionId)/resourceGroups/$($vnet.ResourceGroupName)/providers/Microsoft.Network/virtualNetworks/$($vnetName)"
}
$virtualNetwork = New-AzureRmResource -Location $location -Properties $PropertiesObject -ResourceName "$($webAppName)/$($vnet.Name)" -ResourceType "Microsoft.Web/sites/virtualNetworkConnections" -ApiVersion 2015-08-01 -ResourceGroupName $resourceGroupName -Force
# Now finish joining by getting the VPN package and giving it to the App
Write-Host "Retrieving VPN Package and supplying to App"
$packageUri = Get-AzureRmVpnClientPackage -ResourceGroupName $vnet.ResourceGroupName -VirtualNetworkGatewayName $gateway.Name -ProcessorArchitecture Amd64
# Put the VPN client configuration package onto the App
$PropertiesObject = #{
"vnetName" = $vnet.Name; "vpnPackageUri" = $packageUri
}
New-AzureRmResource -Location $location -Properties $PropertiesObject -ResourceName "$($webAppName)/$($vnet.Name)/primary" -ResourceType "Microsoft.Web/sites/virtualNetworkConnections/gateways" -ApiVersion 2015-08-01 -ResourceGroupName $resourceGroupName -WarningAction silentlyContinue -Force
}
So after 2 weeks of going back and forth with Microsoft (had a really good guy Charles) we managed to find the problem.
When requesting
$packageUri = Get-AzureRmVpnClientPackage -ResourceGroupName $vnet.ResourceGroupName -VirtualNetworkGatewayName $gateway.Name -ProcessorArchitecture Amd64
It was giving me an output of:
"https://mdsbrketwprodsn1prod.blob.core.windows.net/cmakexe/xxx~xxx/amd64/xxxx~xxxx&sp=r&fileExtension=.exe"
For some reason (that Microsoft could explain) why it kept adding in " to the beginning and end of the variable.
I find it odd that it lets the script work with " and allows the WebApps to join to the VPN.
Any why here is the fix which basicly removes the " from the begining and end of $packageUri :
$packageUri = $packageUri.ToString();
$packageUri = $packageUri.Substring(1, $packageUri.Length-2);
So hope that helps someone else out there who is banging there head agaist the same problem.
Here is the complete function if any one is intrested:
function AddExistingVnet{
param(
[string] $subscriptionId,
[string] $resourceGroupName,
[string] $webAppName
)
$Vnet = Get-AzureRmVirtualNetwork | Where-Object {$_.ResourceGroupName -like "*Static*"}
IF($Vnet.Name.count -gt 1) {write-host 'Two or networks have been returned. Unable to continue ' return}
$gatewaySubnet = $vnet.Subnets | Where-Object { $_.Name -eq "GatewaySubnet" }
$vnetName = $vnet.Name
$uriParts = $gatewaySubnet.IpConfigurations[0].Id.Split('/')
$gatewayResourceGroup = $uriParts[4]
$gatewayName = $uriParts[8]
$gateway = Get-AzureRmVirtualNetworkGateway -ResourceGroupName $vnet.ResourceGroupName -Name $gatewayName
$webApp = Get-AzureRmResource -ResourceName $webAppName -ResourceType "Microsoft.Web/sites" -ApiVersion 2015-08-01 -ResourceGroupName $resourceGroupName
$location = $webApp.Location
Write-Host "Creating App association to VNET"
$propertiesObject = #{
"vnetResourceId" = "/subscriptions/$($subscriptionId)/resourceGroups/$($vnet.ResourceGroupName)/providers/Microsoft.Network/virtualNetworks/$($vnetName)"
}
$virtualNetwork = New-AzureRmResource -Location $location -Properties $PropertiesObject -ResourceName "$($webAppName)/$($vnet.Name)" -ResourceType "Microsoft.Web/sites/virtualNetworkConnections" -ApiVersion 2015-08-01 -ResourceGroupName $resourceGroupName -Force
# Now finish joining by getting the VPN package and giving it to the App
Write-Host "Retrieving VPN Package and supplying to App"
$packageUri = Get-AzureRmVpnClientPackage -ResourceGroupName $vnet.ResourceGroupName -VirtualNetworkGatewayName $gateway.Name -ProcessorArchitecture Amd64
$packageUri = $packageUri.ToString();
$packageUri = $packageUri.Substring(1, $packageUri.Length-2);
# Put the VPN client configuration package onto the App
$PropertiesObject = #{
"vnetName" = $vnet.Name; "vpnPackageUri" = $packageUri.ToString()
}
$date = Get-Date -format "HH:mm tt"
New-AzureRmResource -Location $location -Properties $PropertiesObject -ResourceName "$($webAppName)/$($vnet.Name)/primary" -ResourceType "Microsoft.Web/sites/virtualNetworkConnections/gateways" -ApiVersion 2015-08-01 -ResourceGroupName $resourceGroupName -WarningAction silentlyContinue -Force
}
Enjoy
I'm trying to fix my continious deployment scenario and for this to work an Azure website has to exist and be able to swap between the Staging and Production. We want this website to work in the Standard pricing tier.
The script I have at the moment creates a new ResourceGroup, Hosting Plan and after these are created, the website itself. The problem I'm facing is the website is always in Free mode. I should be able to fix this by using the Set-AzureResource cmdlet, but this one is throwing an message telling me I should specify the Location. Problem with this is this specific cmdlet doesn't has a Location parameter.
Set-AzureResource : {
"Error": {
"Code": "LocationRequired",
"Message": "The location property is required for this definition.",
"Target": null,
"Details": null
}
}
This is the (simplified) script I'm using to create all of the resources:
#Create the resource group
New-AzureResourceGroup -Location $location -Name $resourceGroupName -Force
#Create the hosting plan
$hostingPlanParameters = #{"name" = $hostingPlanName; "sku" = "Standard"; "computeMode" = "Standard"; "workerSize" = "0"; "numberOfWorkers" = "1"}
New-AzureResource -ApiVersion 2014-04-01 -Name $hostingPlanName -ResourceGroupName $resourceGroupName `
-ResourceType Microsoft.Web/serverFarms -Location $location `
-PropertyObject $hostingPlanParameters -Verbose -Force
#Create the website
$analyticsSite = #{"sku" = "Standard"; "webHostingPlan" = $hostingplan; "computeMode" = "Standard"; }
New-AzureResource -Name $label -ResourceType Microsoft.Web/sites `
-ResourceGroupName $resourceGroupName -Location $location `
-ApiVersion $apiVersion -PropertyObject $analyticsSite -Force
Set-AzureResource -Name $label -ResourceType Microsoft.Web/sites `
-ResourceGroupName $resourceGroupName -ApiVersion $apiVersion `
-PropertyObject $analyticsSite -Force
I've read the website should inherit the sku of the specified hosting plan, so I should not need to update it. This does not appear to work for my above script. The hosting plan is specified, but the settings aren't inheritted.
The created hosting plan properties look like this:
PropertiesText : {
"name": "devHostingPlanWestEU10",
"sku": "Standard",
"workerSize": 0,
"workerSizeId": 0,
"numberOfWorkers": 1,
"currentWorkerSize": 0,
"currentWorkerSizeId": 0,
"currentNumberOfWorkers": 1,
"status": 0,
"webSpace": "ResourceGroupWestEU10-WestEuropewebspace",
"subscription": "ad7add9b-8b7a-45df-8e95-0e7fccbr78a5",
"adminSiteName": null,
"hostingEnvironment": null,
"maximumNumberOfWorkers": 0,
"planName": null,
"perSiteScaling": null,
"hostingEnvironmentId": null
}
This looks kind of ok to me.
Once the website is created, these properties are printed:
PropertiesText : {
"name": "Testert10",
"state": "Running",
"hostNames": [
"testert10.azurewebsites.net"
],
"webSpace": "ResourceGroupWestEU10-WestEuropewebspace",
...
"repositorySiteName": "Testert10",
"owner": null,
"usageState": 0,
"enabled": true,
...
"computeMode": null,
"serverFarm": "Default1",
"serverFarmId": null,
"lastModifiedTimeUtc": "2015-05-21T11:52:30.773",
"storageRecoveryDefaultState": "Running",
"contentAvailabilityState": 0,
"runtimeAvailabilityState": 0,
"siteConfig": null,
"deploymentId": "Testert10",
"trafficManagerHostNames": null,
"sku": "Free",
"premiumAppDeployed": null,
"scmSiteAlsoStopped": false,
"targetSwapSlot": null,
"hostingEnvironment": null,
"microService": "WebSites",
"gatewaySiteName": null,
"kind": null,
"cloningInfo": null,
"hostingEnvironmentId": null
}
As you can see, the computeMode, serverFarm, hostingEnvironment and sku aren't set with the properties I set in the $analyticsSite object.
Therefore I probably need to update the resource, but this throws the error mentioned above.
I've also tried using the New-AzureWebsite, using Troy Hunt's blogpost as an example. However, this post also relies on using the Set-AzureResource, so I'll fall into the same problem. A different problem with this example is you can't control on which resource group and hosting plan the site is created which will cause a bit of trouble when searching for the site.
This is possible so easily in the new RM cmdlets. Make sure that you have the latest version of Azure PowerShell first.
First create an App service plan that defines the Standard Price tier, then create a web app with the app service plan.
function Create-AppServicePlan()
{
#https://msdn.microsoft.com/en-us/library/mt619306.aspx
$resource = Find-AzureRmResource -ResourceNameContains $ServicePlanName -ResourceGroupNameContains $ResourceGroupName -ResourceType "Microsoft.Web/ServerFarms"
if(!$resource)
{
# Specify the Tier type that you would like
$servicePlan = New-AzureRmAppServicePlan -ResourceGroupName $ResourceGroupName -Name $ServicePlanName -Location $WebAppLocation -Tier Standard -NumberofWorkers 1 -WorkerSize "Small"
}
}
Next create the web app with the app service plan as a parameter.
function Create-AzureRmWebApp()
{
#https://msdn.microsoft.com/en-us/library/mt619250.aspx
$resource = Find-AzureRmResource -ResourceNameContains $WebAppName -ResourceGroupNameContains $ResourceGroupName -ResourceType "Microsoft.Web/sites"
if(!$resource)
{
$webApp = New-AzureRmWebApp -ResourceGroupName $ResourceGroupName -Name $WebAppName -Location $WebAppLocation -AppServicePlan $ServicePlanName
}
}
This is the full working script that is verified working.
$ServicePlanName = "PSScriptAppServicePlann"
$WebAppName = "WebAppByPSlooksCool"
$ResourceGroupName = "MyResourceGroup"
$WebAppLocation = "australiaeast"
$ErrorActionPreference = "Stop"
# Step 1: Create the application service plan
Create-AppServicePlan
# Step 2: Create the web app using the service plan name.
Create-AzureRmWebApp
function Create-AzureRmWebApp()
{
#https://msdn.microsoft.com/en-us/library/mt619250.aspx
$resource = Find-AzureRmResource -ResourceNameContains $WebAppName -ResourceGroupNameContains $ResourceGroupName -ResourceType "Microsoft.Web/sites"
if(!$resource)
{
$webApp = New-AzureRmWebApp -ResourceGroupName $ResourceGroupName -Name $WebAppName -Location $WebAppLocation -AppServicePlan $ServicePlanName
}
}
function Create-AppServicePlan()
{
#https://msdn.microsoft.com/en-us/library/mt619306.aspx
$resource = Find-AzureRmResource -ResourceNameContains $ServicePlanName -ResourceGroupNameContains $ResourceGroupName -ResourceType "Microsoft.Web/ServerFarms"
if(!$resource)
{
# Specify the Tier type that you would like
$servicePlan = New-AzureRmAppServicePlan -ResourceGroupName $ResourceGroupName -Name $ServicePlanName -Location $WebAppLocation -Tier Standard -NumberofWorkers 1 -WorkerSize "Small"
}
}
The problem is that you created your site in a default free App Service Plan (aka Server Farm or Web Hosting Plan - they're all the same thing), called "Default1". Yet the App Service Plan you scaled to the Standard size was a different one called "devHostingPlanWestEU10".
To create a site in a pre-existing App Service Plan use the following command:
(split into multiple lines for readability)
New-AzureResource
-Name <YourSiteName>
-Location "West US"
-ResourceGroupName <YourResourceGroupName>
-ApiVersion 2014-11-01
-ResourceType "Microsoft.Web/sites"
-PropertyObject #{ "serverFarm" = "<Your service plan name>" }
With the New Azure Resource Manager Cmdlet. You can create a new App Service Plan and Pass it to the New-AzureRmWebApp Cmdlet.
New-AzureRmAppServicePlan -Name StdPlan -Location <"Location"> -ResourceGroupName <"ResourceGroupName"> -Tier Standard
New-AzureRmWebApp -ResourceGroupName <"ResourceGroupName"> -Name <"WebAppname"> -Location <"Location"> -AppServicePlan StdPlan
Reference Article: https://learn.microsoft.com/en-us/azure/app-service-web/app-service-web-app-azure-resource-manager-powershell