How to build and deploy SSAS tabular from VSTS through CI/CD locally and to Azure Analysis services - powershell

I am working on a SSAS Tabular project in Visual Studio 2017 where I want to automate build and deploy (and testing) locally and in azure analysis services. The project is connected to an Azure Devops project and it works.
But I am struggling a lot when it comes to build and deploy the project in azure devops. I have followed a blog on this topic (https://notesfromthelifeboat.com/post/analysis-services-1-deployment/) and author has created some powershell scripts that works on my computer when I am running them locally on Windows Powershell ISE. But when I tried to create a build pipeline on devops using the same Powershell file in a Powershell task it fails. I created some variables and set a reference to the powershell file. So far so fine.When I tried to run the build I got an error saying:
Import-Module : The specified module 'SqlServer' was not loaded
because no valid module file was found in any module
It seems that powershell in devops are not able to load the Import-Module -Name SqlServer. I have searched on the net for a solution but nothing has worked so far, and other combination of the ImpI have discovered that there is a small difference in the $env:PSModulePath environment variable between in Powershell ISe and in the Powershell ci-build task, but I am not sure if that is the problem.
If any of you have experience knowing how to solve this issue or have a better solution on how to deploy SSAS Tabular model locally and especially deploying to azure (may be some of you have experience with automation) from build/release.
build setup on devops
Error from running the build
Powershell script
Command: .\deploy_model.ps1 -workspace c:\develop\tabular-automation -environment validation -analysisServicesUsername test_ssas -analysisServicesPassword test_ssas
param(
[Parameter(Mandatory)]
[string]$workspace,
[Parameter(Mandatory)]
[string]$environment,
[Parameter(Mandatory)]
[string]$analysisServicesUsername,
[Parameter(Mandatory)]
[string]$analysisServicesPassword,
[string]$databaseServer = "localhost",
[string]$analysisServicesServer = "localhost"
)
Import-Module -Name SqlServer
$ErrorActionPreference = "Stop"
# Build the model
$msbuild = 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\15.0\Bin\MSBuild.exe'
& "$msbuild" TabularExample.smproj "/p:Configuration=$environment" /t:Clean,Build /p:VisualStudioVersion=14.0
# Copy build outputs and deployment options to deployment directory
$deploymentDir = ".\deployment"
mkdir -Force $deploymentDir
cp "bin\$environment\*.*" $deploymentDir
cp .\deploymentoptions\*.* $deploymentDir
# Update deployment targets with parameters
$template = Get-Content .\deploymentoptions\Model.deploymenttargets
$expandedTemplate = $ExecutionContext.InvokeCommand.ExpandString($template)
$expandedTemplate | Set-Content "$deploymentDir\Model.deploymenttargets"
# Create the deployment script
Microsoft.AnalysisServices.Deployment.exe "$deploymentDir\Model.asdatabase" /s:"$deploymentDir\deploy.log" /o:"$deploymentDir\deploy.xmla" | Out-Default
# Deploy the model
$SECURE_PASSWORD = ConvertTo-SecureString $analysisServicesPassword -AsPlainText -Force
$CREDENTIAL = New-Object System.Management.Automation.PSCredential ($analysisServicesUsername, $SECURE_PASSWORD)
Invoke-ASCmd –InputFile "$workspace\$deploymentDir\deploy.xmla" -Server $analysisServicesServer -Credential $CREDENTIAL

For third party modules, or solution has been to modify the $env:PSModulePath to point to a network location that has the version of the module we want our build agents to run. I use code like below (We also set the PSModulePath to the relative path to our custom modules that are stored in the same repo, but I removed that part of code since you do not state that you have any custom modules)
I like this better than constantly running Install-Module because I have better control over what version of the modules are running, and I do not have to worry about having our build boxes constantly communicating with PowershellGallery
try {
Import-Module SQLServer -Force -ErrorAction Stop
}
catch {
$networkPath = "\\Network path to Modules\"
if (!(Test-Path $networkPath)) {
Write-Error "Can not set env:PSModulePath to the published location on the network" -ErrorAction Stop
}
else {
if (!($env:PSModulePath -like "*;$networkPath*")) {
$env:PSModulePath = $env:PSModulePath + ";$networkPath"
}
}
}
else {
Write-Host "Setting the modulePath to $modulePath"
if (!($env:PSModulePath -like "*;$modulePath*")) {
$env:PSModulePath = $env:PSModulePath + ";$modulePath\"
}
}
Import-Module SQLServer -Force -DisableNameChecking -ErrorAction Stop
}

Related

Install Windows HotFix using Terraform on AWS

I have a very simple PowerShell script that uploads a generated test file to an AWS S3 bucket from a Windows 2008 R2 Datacenter server (clean AWS instance). If I run the script remotely on the server using Terraform (remote-exec provisioner), the script fails on the S3 upload with a StackOverflowException. When I run the script directly on the server, it runs fine and uploads the file.
I've experimented with different sizes for the file and 14.5MB seems to be about the maximum that works before the StackOverflowException occurs. Just about any size works fine when I RDP into the server and run the script directly. I've tested 200MB and it works fine.
Any idea why this is happening or what I can do to fix it? The actual file I need to upload is 50MB.
Here are the essential parts to recreate the problem. terraform.tf file:
resource "aws_instance" "windows" {
count = "1"
ami = "ami-e935fc94" #base win 2008 R2 datacenter
instance_type = "t2.micro"
connection {
type = "winrm"
user = "<username>"
password = "<password>"
timeout = "30m"
}
provisioner "file" {
source = "windows/upload.ps1"
destination = "C:\\scripts\\upload.ps1"
}
provisioner "remote-exec" {
inline = [
"powershell.exe -File C:\\scripts\\upload.ps1"
]
}
}
The PowerShell script is very simple. upload.ps1:
$f = new-object System.IO.FileStream C:\Temp\test.dat, Create, ReadWrite
$f.SetLength(40MB) # change this to 14.5MB and it works!
$f.Close()
Write-S3Object -BucketName "mybucket" -Folder "C:\Temp" -KeyPrefix "20180322" -SearchPattern "*.dat"
The error that I receive when launching the script from Terraform (remote-exec provisioner):
aws_instance.windows (remote-exec): Process is terminated due to StackOverflowException.
Running upload.ps1 from RDP on the server itself works fine, including larger files (tested up to 200MB).
Here is the version information:
Microsoft Windows Server 2008 R2 Datacenter
Powershell Version: 3.0
AWS Tools for Windows PowerShell, Version 3.3.245.0
Amazon Web Services SDK for .NET, Core Runtime Version 3.3.21.15
This problem results from a Windows bug. This is all fine and good for a standard Windows server -- you can patch and move on. But, things are more tricky with AWS automation using Terraform.
The ideal solution would allow 1) use of the base AMI, 2) apply the hotfix to itself, and 3) then run the WinRM remote-exec, all from Terraform. Another solution would be to create an AMI with the hotfix installed and have Terraform generate instances using that AMI. However, then you're stuck maintaining AMIs.
Normally, I grab the Microsoft-provided base AMI using a filter:
data "aws_ami" "windows2008" {
most_recent = true
filter {
name = "virtualization-type"
values = ["hvm"]
}
filter {
name = "name"
values = ["Windows_Server-2008-R2_SP1-English-64Bit-Base*",]
}
owners = ["801119661308", "amazon"]
}
Then I use that AMI to create the AWS instance:
resource "aws_instance" "windows" {
count = "1"
ami = "${data.aws_ami.windows2008.id}"
...
}
But, the base AMI doesn't have the hotfix installed allowing you to avoid this WinRM/Windows bug. This is were it gets tricky.
You can use a userdata script to perform a multi-phase setup. In the first boot of the instance (Phase 1), we'll block the instance so that the remote-exec doesn't come in before we're ready. Then, we'll download and install the hotfix and we'll reboot (thanks to Niklas Akerlund, Micky Balladelli and Techibee). On the second boot (in method described here), we'll unblock the instance (enable WinRM) so that the remote-exec can connect.
Here's my userdata/PowerShell script:
$StateFile = "C:\Temp\userdata_state.txt"
If(-Not (Test-Path -Path $StateFile))
{
# PHASE 1
# Close the instance to WinRM connections until instance is ready (probably already closed, but just in case)
Start-Process -FilePath "winrm" -ArgumentList "set winrm/config/service/auth #{Basic=`"false`"}" -Wait
# Set the admin password for WinRM connections
$Admin = [adsi]("WinNT://./Administrator, user")
$Admin.psbase.invoke("SetPassword", "${tfi_rm_pass}")
# Create state file so after reboot it will know
New-Item -Path $StateFile -ItemType "file" -Force
# Make it so that userdata will run again after reboot
$EC2SettingsFile="C:\Program Files\Amazon\Ec2ConfigService\Settings\Config.xml"
$Xml = [xml](Get-Content $EC2SettingsFile)
$XmlElement = $Xml.get_DocumentElement()
$XmlElementToModify = $XmlElement.Plugins
Foreach ($Element in $XmlElementToModify.Plugin)
{
If ($Element.name -eq "Ec2HandleUserData")
{
$Element.State="Enabled"
}
}
$Xml.Save($EC2SettingsFile)
# Download and install hotfix
# Download self-extractor
$DownloadUrl = "https://hotfixv4.trafficmanager.net/Windows%207/Windows%20Server2008%20R2%20SP1/sp2/Fix467402/7600/free/463984_intl_x64_zip.exe"
$HotfixDir = "C:\hotfix"
$HotfixFile = "$HotfixDir\KB2842230.exe"
mkdir $HotfixDir
(New-Object System.Net.WebClient).DownloadFile($DownloadUrl, $HotfixFile)
# Extract self-extractor
Add-Type -AssemblyName System.IO.Compression.FileSystem
[System.IO.Compression.ZipFile]::ExtractToDirectory($HotfixFile, $HotfixDir)
# Install - NOTE: wusa returns immediately, before install completes, so you must check process to see when it finishes
Get-Item "$HotfixDir\*.msu" | Foreach { wusa ""$_.FullName /quiet /norestart"" ; While (#(Get-Process wusa -ErrorAction SilentlyContinue).Count -ne 0) { Start-Sleep 3 } }
# Reboot
Restart-Computer
}
Else
{
# PHASE 2
# Open WinRM for remote-exec
Start-Process -FilePath "winrm" -ArgumentList "quickconfig -q"
Start-Process -FilePath "winrm" -ArgumentList "set winrm/config/service #{AllowUnencrypted=`"true`"}" -Wait
Start-Process -FilePath "winrm" -ArgumentList "set winrm/config/service/auth #{Basic=`"true`"}" -Wait
Start-Process -FilePath "winrm" -ArgumentList "set winrm/config #{MaxTimeoutms=`"1900000`"}"
}

Azure Data factory continous deployment using powershell

I need to do Continous Integration and Deployment for my Azure Data Factory (ADF).
For this in a Visual Studio solution I have two projects:
one for ADF json files(linked services, datasets etc.).
one for PowerShell script for deploying this ADF into a Azure subscription.
Steps followed
Took MSBUILD of ADF codes and used copy files task to copy into $(Build.ArtifactStagingDirectory).
Used Publish artifacts task to publish in VSTS.
Publish artifacts for PowerShell script as a separate build.
Release
In my release I have a Azure PowerShell script which will invoke these ADF files and deploy it in Azure subscription. I'm using "Build.ArtifactStagingDirectory" for referring my ADF files. But I'm getting the below error -
The term 'Build.ArtifactStagingDirectory' is not recognized as the name of a cmdlet, function, script file, or operable program
foreach($file in Get-ChildItem "$(Build.ArtifactStagingDirectory)" -filter "*LinkedService*")
{
New-AzureRmDataFactoryLinkedService -ResourceGroupName "ADFAutomationResource" -DataFactoryName "ADFCICD190218" -Name $file.BaseName -File $file.FullName -Force | Format-List
}
Let me know how to proceed in this case..as there are no sufficient links explaining this.
Try:
foreach($file in Get-ChildItem $Build.ArtifactStagingDirectory -filter "*LinkedService*")
{
New-AzureRmDataFactoryLinkedService -ResourceGroupName "ADFAutomationResource" -DataFactoryName "ADFCICD190218" -Name $file.BaseName -File $file.FullName -Force | Format-List
}
You're referencing a Build variable in a Release!
I assume you've added your artifacts to the release?
If so; you should be able to refer to them like so:
$(System.DefaultWorkingDirectory)/<Artifact Name>

PowerShell script to deploy ASP.NET Core artifacts to Azure

I need a PowerShell script which will deploy an ASP.NET Core app's artifacts to Azure Web Service. Searching the Internet I managed to find this script:
param($websiteName, $packOutput)
$website = Get-AzureWebsite -Name $websiteName
Stop-AzureWebsite -Name $websiteName
# get the scm url to use with MSDeploy. By default this will be the second in the array
$msdeployurl = $website.EnabledHostNames[1]
$publishProperties = #{'WebPublishMethod'='MSDeploy';
'MSDeployServiceUrl'=$msdeployurl;
'DeployIisAppPath'=$website.Name;
'Username'=$website.PublishingUsername;
'Password'=$website.PublishingPassword
}
$publishScript = "${env:ProgramFiles(x86)}\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\Microsoft\Web Tools\Publish\Scripts\1.2.0\default-publish.ps1"
. $publishScript -publishProperties $publishProperties -packOutput $packOutput
Start-AzureWebsite -Name $websiteName
I am using it the way it is shown on the screenshot:
But...nothing happens as the result of msdeploy command execution: no errors, no data deployed...
So, what is the correct way of deploying ASP.NET Core artifacts with PowerShell?
Visual Studio could generate Windows PowerShell publish script for deploying to a website. The publish script may look like this.
publish script
[cmdletbinding(SupportsShouldProcess=$true)]
param($publishProperties=#{}, $packOutput, $pubProfilePath)
# to learn more about this file visit https://go.microsoft.com/fwlink/?LinkId=524327
try{
if ($publishProperties['ProjectGuid'] -eq $null){
$publishProperties['ProjectGuid'] = 'xxxxxxxx-0260-4800-b864-e9afa92d7fc2'
}
$publishModulePath = Join-Path (Split-Path $MyInvocation.MyCommand.Path) 'publish-module.psm1'
Import-Module $publishModulePath -DisableNameChecking -Force
# call Publish-AspNet to perform the publish operation
Publish-AspNet -publishProperties $publishProperties -packOutput $packOutput -pubProfilePath $pubProfilePath
}
catch{
"An error occurred during publish.`n{0}" -f $_.Exception.Message | Write-Error
}
And a publish module that contains functions that will be used in the scripts. For more information about publishscripts for deploying to a website, please refer to this documentation.

Azure Powershell script fails when run through task scheduler

I have a powershell script that I wrote to backup a local sqlserver to an azure blob. Its based on one I took from MSDN, but I added an extra feature to delete any old backups that are over 30 days old. When I run this as a user, it works fine. When I added this to task scheduler, set to run as me, and I manually ask for it to run, it works fine. (All output is captured in a log file, so I can see that its all working). When run from the task scheduler at night when I'm not logged in (the task scheduler is set to run the script as me) it fails. Specifically, it claims my azure subscription name is not know when I call Set-AzureSubscription. Then, fails when trying to delete the blob with:
Get-AzureStorageBlob : Can not find your azure storage credential. Please set current storage account using "Set-AzureSubscription" or set the "AZURE_STORAGE_CONNECTION_STRING" environment variable.
The script in question:
import-module sqlps
import-module azure
$storageAccount = "storageaccount"
$subscriptionName = "SubName"
$blobContainer = "backup"
$backupUrlContainer = "https://$storageAccount.blob.core.windows.net/$blobContainer/"
$credentialName = "creds"
Set-AzureSubscription -CurrentStorageAccountName $storageAccount -SubscriptionName $subscriptionName
$path = "sqlserver:\sql\servername\SQLEXPRESS\databases"
$alldatabases = get-childitem -Force -path $path | Where-object {$_.name -eq "DB0" -or $_.name -eq "DB1"}
foreach ($db in $alldatabases)
{
Backup-SqlDatabase -BackupContainer $backupUrlContainer -SqlCredential $credentialName $db
}
$oldblobs = Get-AzureStorageBlob -container backup | Where-object { $_.name.Contains("DB") -and (-((($_.LastModified) - $([DateTime]::Now)).TotalDays)) -gt $(New-TimeSpan -Days 30).TotalDays }
foreach($blob in $oldblobs)
{
Write-Output $blob.Name
Remove-AzureStorageBlob -Container "backup" -Blob $blob.Name
}
The backup part of the script works, just not the blob deletion parts. It would appear that something is being done to the environment when I log in that allows the azure powershell scripts to work but that isn't being done when I run the command at night when I'm not logged in.
Any one have any idea what that might be?
Task scheduler is set to run the command with a
Powershell -Command "C:\Scripts\BackupDatabases.ps1" 2>&1 >> "C:\Logs\backup.log"
The Azure PowerShell environment just needs to understand what Azure subscription to work with by default. You probably did this for your own environment, but the task scheduler is running in a different environment.
You just need to add an additional command to the beginning of your script to set the Azure subscription. Something like this:
Set-AzureSubscription -SubscriptionName
The documentation for this command is here. You can also set by SubscriptionID etc. instead of SubscriptionName.
In addition, this article walks through how to connect your Azure subscription to the PowerShell environment.
UPDATE: I messed around and got it working. Try adding a "Select-AzureSubscription" before your Set-AzureSubscription command.
Select-AzureSubscription $subscriptionName
Set-AzureSubscription -SubscriptionName $subscriptionName -CurrentStorageAccountName $storageAccount
The documentation for Select-AzureSubscription is here. If you aren't relying on that storage account being set, you may be able to remove the Set-AzureSubscription command.
I was never able to make the powershell script work. I assume I could have made it work if I had set the credentials in the environment variable, as it said, but I instead wrote a little program to do the work for me.
Visit https://github.com/sillyotter/BackupDBToAzure if you need a tool to backup things to azure blobs and delete old leftover backups.
Thanks for the help!

Download blob from Azure storage with powershell -> LoaderException

I use powershell to download a blob from blobstorage in an Azure startup task. I updated Microsoft.WindowsAzure.Storage library today from 3.0.3.0 to 4.0.1.0 via NuGet.
After the library update files are still downloaded correctly but I get same sort of warning in command window:
'Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.'
function download_from_storage ($container, $blob, $connection, $destination) {
Add-Type -Path ((Get-Location).Path + '\Microsoft.WindowsAzure.Storage.dll')
$storageAccount = [Microsoft.WindowsAzure.Storage.CloudStorageAccount]::Parse($connection)
$blobClient = New-Object Microsoft.WindowsAzure.Storage.Blob.CloudBlobClient($storageAccount.BlobEndpoint, $storageAccount.Credentials)
$container = $blobClient.GetContainerReference($container)
$remoteBlob = $container.GetBlockBlobReference($blob)
$remoteBlob.DownloadToFile($destination + "\" + $blob, [System.IO.FileMode]::OpenOrCreate)
}
$connection_string = 'DefaultEndpointsProtocol=https;AccountName=<AcountName>;AccountKey=<Accountkey>'
# JRE
$jre = 'jre-7u60-windows-x64.exe'
$node = 'node-v0.10.29-x64.msi'
download_from_storage 'java-runtime' $jre $connection_string (Get-Location).Path
download_from_storage 'nodejs' $node $connection_string (Get-Location).Path
Since it is still working I am just clueless why the message occurs in the first place.
This is not exactly an answer to your question but here is a much simpler way of downloading files from blob storage:
$dlPath = "C:\temp\"
$container = "BlobContainer"
Set-AzureSubscription "NameOfYourSubscription" -CurrentStorageAccount "storageAccountName"
Get-AzureStorageContainer $container | Get-AzureStorageBlob |
Get-AzureStorageBlobContent -Destination $container
You can do this by installing Azure PowerShell itself in the startup task and then execute the download Azure blob cmdlet. Here are rouphly the steps
Installing Azure PowerShell automatically
Create new service project (New-AzureServiceProject)
Execute Add-AzureWebRole
Change the cscfg to use osFamily=3 (to use new PS version which is compatible with Azure PS)
Copy Azure PowerShell MSI under WebRole1\bin directory
Edit WebRole1\startup.cmd to include this line msiexec /i AzurePowerShell.msi /quiet
Authenticating Azure PowerShell so it can execute cmdlets (if you want to use storage cmdlets only you can ignore this step and pass your storage key/name when executing the Get-AzureStorageBlobContent cmdlet)
Copy a latest publish settings file (myPublishSettings.publishsettings) inside WebRole1\bin folder
Edit WebRole1\startup.cmd to include this line after the one added before: PowerShell.exe –Command “Import-AzurePublishSettingsFile .\myPublishSettings.publishsettings)