Capture errors within ForEach-Object -Parallel in Powershell 7 - powershell

I am trying to execute Powershell script (7.0) file using Powershell 7-Preview which iterates through all the databases, update them using DACPAC file and execute other SQL Server scripts in all the DBs.
It works fine when there is no errors, however, in case of an error while executing Dacpac file
It gives below error and stops executing the script further.
##[error] Could not deploy package.
##[error]PowerShell exited with code '1'.
Any pointer on how we can catch the errors gracefully in PowerShell within the Parallel statement and let script to be continued for other databases? Try-Catch block does not seem to be working here.
I am new to PowerShell. This PowerShell script is a part of DevOps release pipeline.
#.. Variables are defined here ..#
[string[]]$DatabaseNames = Invoke-Sqlcmd #GetDatabasesParams | select -expand name
[int]$ErrorCount = 0
$DatabaseNames | ForEach-Object -Parallel {
try
{
echo "$_ - Updating database using DACPAC."
dir -Path "C:\Program Files (x86)\Microsoft Visual Studio*" -Filter "SqlPackage.exe" -Recurse -ErrorAction SilentlyContinue | %{$_.FullName} {$using:SqlPackagePath /Action:Publish /tu:$using:DatabaseUsername /tp:$using:DatabasePassword /tsn:$using:ServerInstance /tdn:"$_" /sf:using:$DacpacLocation /p:BlockOnPossibleDataLoss=False}
echo "$_ - Updating the scripts."
$OngoingChangesScriptParams = #{
"Database" = "$_"
"ServerInstance" = "$ServerInstance"
"Username" = "$DatabaseUsername"
"Password" = "$DatabasePassword"
"InputFile" ="$SystemWorkingDir\$OngoingChangesScriptLocation"
"OutputSqlErrors" = 1
"QueryTimeout" = 9999
"ConnectionTimeout" = 9999
}
Invoke-Sqlcmd #OngoingChangesScriptParams
}
catch {
$ErrorCount++
echo "Internal Error. The remaining databases will still be processed."
echo $_.Exception|Format-List -force
}
}```
Logs-
2020-09-17T19:21:59.3602523Z *** The column [dbo].[FileJob].[GMTOffset] on table [dbo].[FileJob] must be added, but the column has no default value and does not allow NULL values. If the table contains data, the ALTER script will not work. To avoid this issue you must either: add a default value to the column, mark it as allowing NULL values, or enable the generation of smart-defaults as a deployment option.
2020-09-17T19:21:59.4253722Z Updating database (Start)
2020-09-17T19:21:59.4274293Z An error occurred while the batch was being executed.
2020-09-17T19:21:59.4280337Z Updating database (Failed)
2020-09-17T19:21:59.4330894Z ##[error]*** Could not deploy package.
2020-09-17T19:22:00.3399607Z ##[error]PowerShell exited with code '1'.
2020-09-17T19:22:00.7303341Z ##[section]Finishing: Update All Companies

Related

How to prevent multiple instances of the same PowerShell 7 script?

Context
On a build server, a PowerShell 7 script script.ps1 will be started and will be running in the background in the remote computer.
What I want
A safenet to ensure that at most 1 instance of the script.ps1 script is running at once on the build server or remote computer, at all times.
What I tried:
I tried meddling with PowerShell 7 background jobs (by executing the script.ps1 as a job inside a wrapper script wrapper.ps1), however that didn't solve the problem as jobs do not carry over (and can't be accessed) in other PowerShell sessions.
What I tried looks like this:
# inside wrapper.ps1
$running_jobs = $(Get-Job -State Running) | Where-Object {$_.Name -eq "ImportantJob"}
if ($running_jobs.count -eq 0) {
Start-Job .\script.ps1 -Name "ImportantJob" -ArgumentList #($some_variables)
} else {
Write-Warning "Could not start new job; Existing job detected must be terminated beforehand."
}
To reiterate, the problem with that is that $running_jobs only returns the jobs running in the current session, so this code only limits one job per session, allowing for multiple instances to be ran if multiple sessions were mistakenly opened.
What I also tried:
I tried to look into Get-CimInstance:
$processes = Get-CimInstance -ClassName Win32_Process | Where-Object {$_.Name -eq "pwsh.exe"}
While this does return the current running PowerShell instances, these elements carry no information on the script that is being executed, as shown after I run:
foreach ($p in $processes) {
$p | Format-List *
}
I'm therefore lost and I feel like I'm missing something.
I appreciate any help or suggestions.
I like to define a config path in the $env:ProgramData location using a CompanyName\ProjectName scheme so I can put "per system" configuration.
You could use a similar scheme with a defined location to store a lock file created when the script run and deleted at the end of it (as suggested already within the comments).
Then, it is up to you to add additional checks if needed (What happen if the script exit prematurely while the lock is still present ?)
Example
# Define default path (Not user specific)
$ConfigLocation = "$Env:ProgramData\CompanyName\ProjectName"
# Create path if it does not exist
New-Item -ItemType Directory -Path $ConfigLocation -EA 0 | Out-Null
$LockFilePath = "$ConfigLocation\Instance.Lock"
$Locked = $null -ne (Get-Item -Path $LockFilePath -EA 0)
if ($Locked) {Exit}
# Lock
New-Item -Path $LockFilePath
# Do stuff
# Remove lock
Remove-Item -Path $LockFilePath
Alternatively, on Windows, you could also use a scheduled task without a schedule and with the setting "If the task is already running, then the following rule applies: Do not start a new instance". From there, instead of calling the original script, you call a proxy script that just launch the scheduled task.

Gitlab CI/CD validate powershell script

I am distributing several powershell scripts to various machines and want to make sure it is executable and does not contain any errors before uploading the new version to our distribution system.
How can I solve this with Gitlab .gitlab-ci.yml?
I have found a solution for my issue. This step in the build process checks the Powershell script and aborts the build if there are any errors.
gitlab-ci.yml
stages:
- validate
validate_script:
stage: validate
image:
name: "mcr.microsoft.com/powershell:ubuntu-20.04"
script:
- pwsh -File validate.ps1
tags:
- docker
validate.ps1
Write-Host "Install PSScriptAnalyzer"
Install-Module PSScriptAnalyzer -Scope CurrentUser -Confirm:$False -Force
Write-Host "Check mypowershell.ps1"
$result = Invoke-ScriptAnalyzer -Path 'src/mypowershell.ps1'
if ($result.Length -gt 0) {
Write-Host "Script Error"
Write-Host ($result | Format-Table | Out-String)
exit 1;
}
Not sure if this is what you are looking for or if it applies at all to .yml files as I'm not familiar with them, but for powershell script files you can use the creation of [scriptblock] to check for syntax and other compile errors. You can create a function with this code and pass in powershell script files to it which can take the code, try to compile it, and return any exceptions encountered.
# $code = Get-Content -Raw $file
$code = #"
while {
write-host "no condition set on while "
}
do {
Write-Host "no condition set on unitl "
} until ()
"#
try {
[ScriptBlock]::Create($code)
}
catch {
$_.Exception.innerexception.errors
}
Output
Extent ErrorId Message IncompleteInput
------ ------- ------- ---------------
MissingOpenParenthesisAfterKeyword Missing opening '(' after keyword 'while'. False
MissingExpressionAfterKeyword Missing expression after 'until' in loop. False
I also recommend looking into Pester for testing scripts. If you are unfamiliar it is a testing and mocking framework for Powershell code

Azure Pipelines - logging commands from SQL script

I am trying to log some messages from a TSQL script running via Azure Pipelines, for instance, before creating a table we check if table already exists and if so we simply print a message and skip table creation...
there are good articles explaining how to access Azure Pipelines Logging Commands from BASH or PowerShell, for instance this article: https://learn.microsoft.com/en-us/azure/devops/pipelines/scripts/logging-commands?view=azure-devops&tabs=bash
but how to output messages to the pipeline logs from within TSQL statement itself?
I will try with RAISERROR ( e.g. RAISERROR('Table [dbo].[ReportHistory] already exists!', 0, 1) WITH NOWAIT; ) hopefully works better than PRINT command, has anyone had similar issue and how did he resolve it?
You can run your scripts through PowerShell Invoke-Sqlcmd with -Verbose key. Here is small example for PowerShell task:
$server = "$(servername)"
$dbname = "$(dbname)"
$u = "$(username)"
$p = "$(password)"
$filename = "testfile.sql"
$filecontent = "RAISERROR('Table [dbo].[ReportHistory] already exists!', 0, 1) WITH NOWAIT;`r`nGO`r`n"
Set-Content -Path $filename -Value $filecontent
Write-Host '##[command] Executing file... ', $filename
#Execution of SQL packet
try
{
Invoke-Sqlcmd -InputFile "$filename" -ServerInstance $server -Database $dbname -Username "$u" -Password "$p" -QueryTimeout 36000 -Verbose
}
catch
{
Write-Host "##[error]" $Error[0]
Write-Host "##[error]----------\n"
}
Result:
You can use Azure Pipelines Logging Commands together with PRINT and RAISERROR commands.
The logging commands syntax ##vso[task..] is the reserved keywords in Azure devops piplines. When ##vso[task..] is found in the tasks' output stream, Azure devops pipeline will execute the logging commands.
So that you can output messages to the pipeline logs from within TSQL statement using logging commands with PRINT or RAISERROR. See below example:
PRINT N'##vso[task.logissue type=warning]Table already exists.';
RAISERROR('##vso[task.logissue type=warning]Table [dbo].[ReportHistory] already exists!',0,1);
See below output messages in pipeline logļ¼š

How to determine if Write-Host will work for the current host

Is there any sane, reliable contract that dictates whether Write-Host is supported in a given PowerShell host implementation, in a script that could be run against any reasonable host implementation?
(Assume that I understand the difference between Write-Host and Write-Output/Write-Verbose and that I definitely do want Write-Host semantics, if supported, for this specific human-readable text.)
I thought about trying to interrogate the $Host variable, or $Host.UI/$Host.UI.RawUI but the only pertinent differences I am spotting are:
in $Host.Name:
The Windows powershell.exe commandline has $Host.Name = 'ConsoleHost'
ISE has $Host.Name = 'Windows PowerShell ISE Host'
SQL Server Agent job steps have $Host.Name = 'Default Host'
I have none of the non-Windows versions installed, but I expect they are different
in $Host.UI.RawUI:
The Windows powershell.exe commandline returns values for all properties of $Host.UI.RawUI
ISE returns no value (or $null) for some properties of $Host.UI.RawUI, e.g. $Host.UI.RawUI.CursorSize
SQL Server Agent job steps return no values for all of $Host.UI.RawUI
Again, I can't check in any of the other platforms
Maintaining a list of $Host.Name values that support Write-Host seems like it would be bit of a burden, especially with PowerShell being cross-platform now. I would reasonably want the script to be able to be called from any host and just do the right thing.
Background
I have written a script that can be reasonably run from within the PowerShell command prompt, from within the ISE or from within a SQL Server Agent job. The output of this script is entirely textual, for human reading. When run from the command prompt or ISE, the output is colorized using Write-Host.
SQL Server jobs can be set up in two different ways, and both support capturing the output into the SQL Server Agent log viewer:
via a CmdExec step, which is simple command-line execution, where the Job Step command text is an executable and its arguments, so you invoke the powershell.exe executable. Captured output is the stdout/sterr of the process:
powershell.exe -Command x:\pathto\script.ps1 -Arg1 -Arg2 -Etc
via a PowerShell step, where the Job Step command text is raw PS script interpreted by its own embedded PowerShell host implementation. Captured output is whatever is written via Write-Output or Write-Error:
#whatever
Do-WhateverPowershellCommandYouWant
x:\pathto\script.ps1 -Arg1 -Arg2 -Etc
Due to some other foibles of the SQL Server host implementation, I find that you can emit output using either Write-Output or Write-Error, but not both. If the job step fails (i.e. if you throw or Write-Error 'foo' -EA 'Stop'), you only get the error stream in the log and, if it succeeds, you only get the output stream in the log.
Additionally, the embedded PS implementation does not support Write-Host. Up to at least SQL Server 2016, Write-Host throws a System.Management.Automation.Host.HostException with the message A command that prompts the user failed because the host program or the command type does not support user interaction.
To support all of my use-cases, so far, I took to using a custom function Write-Message which was essentially set up like (simplified):
$script:can_write_host = $true
$script:has_errors = $false
$script:message_stream = New-Object Text.StringBuilder
function Write-Message {
Param($message, [Switch]$iserror)
if ($script:can_write_host) {
$private:color = if ($iserror) { 'Red' } else { 'White' }
try { Write-Host $message -ForegroundColor $private:color }
catch [Management.Automation.Host.HostException] { $script:can_write_host = $false }
}
if (-not $script:can_write_host) {
$script:message_stream.AppendLine($message) | Out-Null
}
if ($iserror) { $script:has_errors = $true }
}
try {
<# MAIN SCRIPT BODY RUNS HERE #>
}
catch {
Write-Message -Message ("Unhandled error: " + ($_ | Format-List | Out-String)) -IsError
}
finally {
if (-not $script:can_write_host) {
if ($script:has_errors) { Write-Error ($script:message_stream.ToString()) -EA 'Stop' }
else { Write-Output ($script:message_stream.ToString()) }
}
}
As of SQL Server 2019 (perhaps earlier), it appears Write-Host no longer throws an exception in the embedded SQL Server Agent PS host, but is instead a no-op that emits nothing to either output or error streams. Since there is no exception, my script's Write-Message function can no longer reliably detect whether it should use Write-Host or StringBuilder.AppendLine.
The basic workaround for SQL Server Agent jobs is to use the more-mature CmdExec step type (where Write-Output and Write-Host both get captured as stdout), but I do prefer the PowerShell step type for (among other reasons) its ability to split the command reliably across multiple lines, so I am keen to see if there is a more-holistic, PowerShell-based approach to solve the problem of whether Write-Host does anything useful for the host I am in.
Just check if your host is UserInteractive or an service type environment.
$script:can_write_host = [Environment]::UserInteractive
Another way to track the output of a script in real time is to push that output to a log file and then monitor it in real time using trace32. This is just a workaround, but it might work out for you.
Add-Content -Path "C:\Users\username\Documents\PS_log.log" -Value $variablewithvalue

Newman with AzureDevOps + Powershell

Good morning everyone
I am trying integrate postman tests with AzureDevops Release pipeline.
I have two steps:
First step is to install newman
Second step is to run collection scripts with newman run comand
The second step looks like:
try
{
$testFiles = Get-ChildItem *.postman_collection.json -Recurse
$environmentFile = Get-ChildItem *staging.postman_environment.json -Recurse
Write-Host $testFiles.Count files to test
foreach ($f in $testFiles)
{
$environment = $environmentFile[0].FullName
Write-Host running file $f.FullName
Write-Host usting environment $environment
$collection = $f.FullName
$resultFile = "Results\" + $f.BaseName + ".xml"
Write-Host running $collection
Write-Host will create $resultFile
$(newman run $collection -e $environment -r junit --reporter-junit-export $resultFile)
}
}
catch
{
Write-Host "Exception occured"
Write-Host $_
}
Above step do not work as expected. In the release log I can see the both messages like:
Write-Host running $collection
Write-Host will create $resultFile
However the line
$(newman run $collection -e $environment -r junit --reporter-junit-export $resultFile)
is not being executed.
I did the same on my local machine and the command is working. However the bad thing is the try catch block is not working and only I can see as the result is :
2019-11-22T15:11:23.8332717Z ##[error]PowerShell exited with code '1'.
2019-11-22T15:11:23.8341270Z ##[debug]Processed: ##vso[task.logissue type=error]PowerShell exited with code '1'.
2019-11-22T15:11:23.8390876Z ##[debug]Processed: ##vso[task.complete result=Failed]Error detected
2019-11-22T15:11:23.8414283Z ##[debug]Leaving D:\a\_tasks\PowerShell_e213ff0f-5d5c-4791-802d-52ea3e7be1f1\2.151.2\powershell.ps1.
Do anyone know how to get real error or had experience with newman testing in AzureDevOps ?
When you run those above scripts in VSTS, please remove $() in the newman run line:
newman run $collection -e $environment -r junit --reporter-junit-export $resultFile
Then the script can be run very successfully.
I think you have known that for powershell command line, there will no result displayed in the powershell command line interface in the even if the newman run command has been ran succeed. So, there will no any directly message displayed in the log to let you know whether it is succeed. To confirm this in VSTS, you could check the agent cache if you are using private agent: