I have a Pull Request validation build running for different branches, i.e. several concurrent instances of the build is a very common thing.
One of the things the build does is install a module. Now I could modify the profile on the build agent and install module from there, but I want to avoid any extra build agent configuration. So, my build installs a module in the current user scope.
I noticed that Install-Module does not seem to be safe when invoked concurrently - it may fail with all kinds of different and weird error messages.
Now I solved this with a named mutex acquired before and released after, but this causes abysmal performance - the code sometimes waits for 30 seconds and more.
So, how to solve this problem? How to install a powershell module concurrently, but safely and with good performance?
EDIT 1
It is frustrating. I am trying to trace the concurrent installs using Set-PSDebug -Trace 2, but apparently Install-Module has a lot of Write-Debug invocations calling to functions which are themselves not safe for concurrent execution! So trying to trace actually worsens matters.
Apparently, Install-Module is totally NOT SAFE to run during a build where multiple builds run on the same agent. Looks like using a named Mutex is the safest approach.
EDIT 1
In a multi-threading environments invoking the following commands is not safe without explicit mutex:
Install-Module
Import-Module
Get-PSRepository without arguments
Maybe more. In my code I invoke all the three commands, and I discovered that all of them together must be in the same mutex, i.e. these combinations do not work:
Not working #1
$mtx.WaitOne()
try
{
Install-Module ...
}
finally
{
$mtx.ReleaseMutex()
}
Import-Module ...
Get-PSRepository ...
Not working #2
$mtx.WaitOne()
try
{
Install-Module ...
Import-Module ...
}
finally
{
$mtx.ReleaseMutex()
}
Get-PSRepository
The only safe option appears to be this one:
$mtx.WaitOne()
try
{
Install-Module ...
Import-Module ...
Get-PSRepository
}
finally
{
$mtx.ReleaseMutex()
}
Which is surprising, because I do not expect Install-Module or Import-Module to affect Get-PSRepository, yet somehow they do:
ParameterBindingException: A parameter cannot be found that matches parameter name 'Provider'.
at Get-PSRepository<Process>, C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1: line 4496
at Use-ModuleFB22C60E, C:\Users\mkharitonov\AppData\Local\Temp\fb22c60e-a0c5-48b3-953a-0b580c6a2f5e\m_deadbeef_.ps1: line 167
at <ScriptBlock>, <No file>: line 4
Related
I use Powershell 5.1 and we often install modules from our internal module repository hosted on our on-prem Azure Artifacts (we use Azure DevOps Server 2019).
The problem is that it is very slow. Takes regularly over 10 seconds to load one module. And this is not the network, which is pretty fast. It is the Install-Module internals.
I tried running Set-PSDebug -Trace 2 from within an Azure DevOps build in order to get line timestamps, but it is useless. For example, observe this output snippet:
2020-06-29T04:20:40.6944925Z DEBUG: 267+ switch ( >>>> $MsgID)
2020-06-29T04:20:40.6957451Z DEBUG: ! SET $switch = ''.
2020-06-29T04:20:40.6972578Z DEBUG: 290+ >>>> }
2020-06-29T04:20:40.6986528Z DEBUG: ! SET $switch = ''.
2020-06-29T04:20:40.6998323Z DEBUG: 232+ >>>> }
2020-06-29T04:20:48.3791151Z DEBUG: 220+ $script:PackageManagementInstallModuleMessageResolverScriptBlock = >>>> {
2020-06-29T04:20:48.3808676Z DEBUG: ! CALL function '<ScriptBlock>' (defined in file 'C:\Program
2020-06-29T04:20:48.3811147Z Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1')
2020-06-29T04:20:48.3822332Z DEBUG: 222+ >>>> $PackageTarget =
2020-06-29T04:20:48.3824673Z $LocalizedData.InstallModulewhatIfMessage
It show 8 seconds pause, but the place where it shows it does not make any sense.
So, my question is this - why is it so slow? Is there a way to profile it reliably?
EDIT 1
Have just installed PS Core 7 - the same lousy performance for Install-Module. My version of PowershellGet is:
C:\> Get-Module PowershellGet | Select Version
Version
-------
2.2.4
C:\>
EDIT 2
Found this page - https://learn.microsoft.com/en-us/powershell/scripting/gallery/how-to/working-with-packages/manual-download?view=powershell-7 It explicitly warns against simulating Install-Module with nuget, even though it explains how to do it. I would like to understand more about the implications of using nuget instead of Install-Module, besides it working 5 times faster (on average).
EDIT 3
The modules are not signed. We are talking about our internal modules. But installing modules from PSGallery, like Az.Accounts, Az.Storage and Az.Sql takes about the same time. When our build needs to make sure 5 modules are installed it takes easily a minute. On another note, Install-Module is not concurrency safe, so when our builds were running it bare we were facing all kinds of weird errors. They went away when we introduced explicit named mutex around Install-Module. Needless to say, it did not contribute to performance.
While this doesn't answer your "why", you might like to look at JustinGrote's high performance Powershell Gallery Module Installer:
.SYNOPSIS
High Performance Powershell Module Installation
.DESCRIPTION
This is a proof of concept for using the Powershell Gallery OData
API and HTTPClient to parallel install packages
It is also a demonstration of using async tasks in powershell
appropriately. Who says powershell can't be fast?
This drastically reduces the bandwidth/load against Powershell
Gallery by only requesting the required data
It also handles dependencies (via Nuget), checks for existing
packages, and caches already downloaded packages
.NOTES
THIS IS NOT FOR PRODUCTION, it should be considered "Fragile" and
has very little error handling and type safety
It also doesn't generate the PowershellGet XML files currently, so
PowershellGet will see them as "External" modules
It is indeed much faster.
I'm installing SQL Data Tools via a PowerShell script. I run my script, but the final part where the Data Tools are installed fails (inside of the SQL installer window). If I run the script without that part, and install Data Tools manually it works.
The error is:
VS Shell installation has failed with exit code -2147205120.
The parts before this install .NET and SQL Server Management Studio. I don't think they're relevant to my issue, but I will post that part if requested. Here are the relevant parts. The first try block installs SQL SP1 (removed now for readability), the second installs Data Tools and SNAC_SDK.
try
{
Write-Host "Lauching SQL Server Data Tools install ..."
& "\\mynetworkpath\SSDTBI_x86_ENU.exe" "/ACTION=INSTALL" "/FEATURES=SSDTBI,SNAC_SDK" "/Q" "/IACCEPTSQLSERVERLICENSETERMS"
Write-Host "Installer launched ..."
}
catch
{
Write-Host "SQL Server Data Tools installation failed"
exit
}
I have tried juggling around the arguments for the Data Tools install part, and playing with the -wait verb to make sure SP1 is done for sure, but no luck.
EDIT: Per Matt's suggestion I added /NORESTART to my argument list, but now it doesn't install anything, and doesn't error either...
EDIT: Added updated code with quoted arguments. Still doesn't work, but I think it's closer than it was originally.
I think the comma in the arguments is the culprit here because powershell interprets entities separated by comma as an array.
You can see how parameters get passed with this little hack
& { $args } /ACTION=INSTALL /FEATURES=SSDTBI,SNAC_SDK /Q /IACCEPTSQLSERVERLICENSETERMS
which gives
/ACTION=INSTALL
/FEATURES=SSDTBI
SNAC_SDK
/Q
/IACCEPTSQLSERVERLICENSETERMS
To get rid of this problem you need to quote at least the FEATURES argument. I usually quote everything in those cases, just to be consistent, so
& { $args } "/ACTION=INSTALL" "/FEATURES=SSDTBI,SNAC_SDK" "/Q" "/IACCEPTSQLSERVERLICENSETERMS"
gives you the wanted parameters:
/ACTION=INSTALL
/FEATURES=SSDTBI,SNAC_SDK
/Q
/IACCEPTSQLSERVERLICENSETERMS
Update: Many installers return immediately after they have been called while the install process still runs in the background, which can be a bugger when the rest of the script depends on the install.
There are several methods to make powershell wait for a process exit. One of the shortest is to use Out-Null like this:
& "\\mynetworkpath\SSDTBI_x86_ENU.exe" "/ACTION=INSTALL" "/FEATURES=SSDTBI,SNAC_SDK" "/Q" "/IACCEPTSQLSERVERLICENSETERMS" | Out-Null
You may also want to look at $? or $LASTEXITCODE afterwards to check for errors.
Just installed management framework 5 production preview, my profile won't load anymore with the following error
Get-Module : Cannot load the module 'UserInfo.psm1' because the module nesting limit has been exceeded. Modules can only be nested
to 10 levels. Evaluate and change the order in which you are loading modules to prevent exceeding the nesting limit, and then try running your script again.
Is there a way to get more details?
I tried to trace execution with set-psdebug -trace 2, but I can't see the problem...
Found it, I had same modules loaded from .psd1 RequiredModules = #('coresmac','activedirectory') and from the .psm1 #requires -modules ActiveDirectory,userinfo .
As the requires instruction appears as a comment it was quite easy to miss it ...
I have found that module auto-loading is unreliable. #briantist recommends explicitly importing required modules in finished code. This seems like good advice to me. The problem is I haven't found a reliable way to ensure that all required modules are explicitly imported.
Example
Supposed I have three modules with these dependencies:
m1.psm1 (depends on nothing)
m2.psm1 (calls functions in m1)
Suppose also that each of these modules has a set of automated tests that achieve 100% code coverage.
Running the tests for m2 only alerts you to a missing call to Import-Module m1 in the rare occurrence that PowerShell's auto-loading happens to fail to load m2 at the specific time you run the test. If you have several 10s of .psm1 files, the odds of missing a required Import-Module increases dramatically.
Suppose I mistakenly omit Import-Module m1 in m2.psm1 (after all, it passed all the automated tests) and put these modules into production. Each time functions in m2 are executed there is a small chance that m1 will not auto-load. When m1 does not successfully auto-load, the script fails to execute correctly that time, but may well execute correctly the time before and the time after. This will make it rather difficult to diagnose the cause of problem.
How do you ensure that all modules a script or module depends on have been explicitly imported?
Clarification: The modules I am developing are mostly automations that will ultimately run unattended on a server. Accordingly, I need to ensure that the functions in the modules will execute quickly and correctly once they are deployed on a machine other than my development workstation.
It is, of course, possible to import all modules you know you need explicitly when you start the powershell session. However, the auto-loading mechanism still masks the case where you forgot to explicitly load a module you depend on.
You can also blindly load all modules at startup (I do this at some stages during development), but depending on that means a start-up time on the order of 10s of seconds that grows with the size of your module library. That startup time will likely become a problem because it will be incurred each time a script is executed.
This is how i personally do it, for comparison, in my profile i have
# Get the contents of the module folder
$modules = Get-ChildItem "$PSScriptRoot\modules" -Directory
# For each item
ForEach($module in $modules)
{
Try
{
# Try loading the module in the folder
Import-Module $module.FullName
}
Catch
{
# Oh darn
Write-Warning "Could not load $module"
}
}
And in my module files i also have at the very top
#requires -version 3
#requires -module MyModule
Get-Module MyModule | Select -ExpandProperty RequiredModules | Import-Module
I finally settled on the following strategy to detect any modules that have not been explicitly imported:
Use $PSModuleAutoLoadingPreference='none' to stop the powershell environment from auto-loading anything.
Never import any modules from a text fixture.
Remove as many modules as possible prior to running each test.
Pester Conventions
To achieve the above I settled on the following convention for Pester test fixtures:
Remove-egModules
$Global:mut = 'egMyModule'
Import-Module $mut
InModuleScope $mut {
Describe{
BeforeEach{ Remove-egModules -Except $mut }
It 'does something useful' {
...
}
}
}
Remove-Variable 'mut' -Scope Global
Furthermore, any "helper" modules required by the test fixture but not by the module under test must be imported in the module under test despite that they are required only in the test fixture. Why? Suppose you import a helper module inside the test fixture and that helper module also happens to be used by the module under test. The tests will succeed even if you have not explicitly imported the helper module inside the module under test. When the test fixture imports a module, it is also made available to the module under test.
Remove-egModules
I determined that I needed to repeatedly remove all but the very minimum of modules when running tests. This is because modules seem to remain loaded for an entire powershell session. To achieve that, I use this function:
function Remove-egModules
{
[CmdletBinding()]
param
(
# this module does not get removed
$Except
)
process
{
Get-Module |
? { $_.Name -ne 'egRemoveModule' } | # the module where this function is
? {
... # test for any other modules that should be removed
} |
? { $_.Name -ne $Except } |
Remove-Module |
Out-Null
}
}
Vote++ for Bluecakes' answer. My preference, though, is to control the list of modules using an array, that is, don't just load the modules found in the module path. Personal preference, I know. Also, I recommend dropping the modules prior to importing - otherwise PS will cache them for you in the current session, making development awkward (i.e.: you modify a module, but it doesn't get reloaded).
Generally one can use $ErrorActionPreference to stop the execution of a PS script on error as in the example below that uses an invalid subversion command. The script exits after the first command.
$ErrorActionPreference='Stop'
svn foo
svn foo
Trying the same thing with the maven command (mvn.bat) executes both commands.
$ErrorActionPreference='Stop'
mvn foo
mvn foo
Initially I suspected that mvn.bat does not set an error code and PS just doesn't see the error. However, $? is set properly as demonstrated by the following, when PS exits after the first error:
mvn foo
if (-not $?) {exit 1}
mvn foo
So here's my question: Given that both svn.exe and mvn.bat set an error code on failure, why does the PS script not stop after the mvn error. I find it a lot more convenient to set "$ErrorActionPreference=Stop" globally rather than doing "if (-not $?) {exit 1}" after each command to terminate on error.
Not all command line programs provide errors in the same way. Some set an exit code. Some don't. Some use the error stream, some don't. I've seen some command line programs actually output everything to error, and always output non-zero return codes.
So there's not a real safe guess one could ever make as to it having run successfully, and therefore it's next to impossible to codify that behavior into PowerShell.
$errorActionPreference will actually stop a script whenever a .exe writes to the error stream, but many .exes will write regular output to the console and never populate the error stream.
But it will not reliably work. Neither, for that matter, will $?. If an exe returns 0 and doesn't write to error $? will be true.
While it might be a pain in the butt to deal with each of these individual .exes and their errors in PowerShell, it's a great example of why PowerShell's highly structured Output/Error/Verbose/Warning/Debug/Progress streams and consistent error behavior in Cmdlets beats out plain old .EXE tools.
Hope this Helps
$ErrorActionPreference controls PowerShell's behavior with regard to commandlets and PowerShell functions -- it is not sensitive to exit codes of "legacy" commands. I'm still looking for a work-around for this design decision myself -- for this issue, refer to this thread: How to stop a PowerShell script on the first error?. You'll probably conclude that this custom "exec" function is the best solution: http://jameskovacs.com/2010/02/25/the-exec-problem/
With regard to the stop-on-stderr behavior, I can't reproduce this behavior in PowerShell 3:
Invoke-Command
{
$ErrorActionPreference='Stop'
cmd /c "echo hi >&2" # Output to stderr (this does not cause termination)
.\DoesNotExist.exe # Try to run a program that doesn't exist (this throws)
echo bye # This never executes
}
Update: The stop-on-stderr behavior appears to take effect when using PS remoting.