How to handle (avoid): "Fail to create runspace because you have exceeded your budget to create runspace." - powershell

I have HTTP triggered Azure Function App on PowerShell Core stack. Script is parsing the body of the request, assuming that everything is ok, it connects to Exchange Online and then executes 2 cmdlets to create MailContact type of contact. At the end it disconnects from Exchange Online. I have console app that is executing POST requests passing JSON data for one contact in the body. Requests are executed in a for-each loop and after 5th successful requests I get runspace exceeded budget error.
some code snippets from the script
...
try {
Connect-ExchangeOnline -CertificateThumbprint $thumb -AppId $appId -Organization $org -Showbanner:$false -CommandName Get-Contact,Get-MailContact,New-MailContact,Set-Contact,Set-MailContact,Remove-MailContact
New-MailContact -ErrorAction stop #p | Out-Null
Set-Contact -ErrorAction stop #parameters | Out-Null
}
catch {
...
}
finally {
Disconnect-ExchangeOnline -Confirm:$false -InformationAction Ignore -ErrorAction SilentlyContinue
Get-PSSession | Remove-PSSession
}
What I tried (without success):
relaxation for Exchange Online throttling policy (https://www.michev.info/Blog/Post/3205/self-service-powershell-throttling-policy-relaxation-for-exchange-online)
setting different environmental variables (like PSWorkerInProcConcurrencyUpperBound and FUNCTIONS_WORKER_PROCESS_COUNT)
What worked: having additional Function App and then cycle every 5 requests between the two.
Additional information that might help:
PSWorkerInProcConcurrencyUpperBound = 1000
FUNCTIONS_WORKER_PROCESS_COUNT = 10
Function runtime version = ~4
PowerShell Core Version = 7
Platform = 64Bit
Plan type = Consumption (Serverless)
On addition, it takes around 7-8 sec from sending request till I get the response back. Connecting to Exchange Online takes a lot of time.
Any help or hint how to solve the runspace budget error ?

a dirty workaround would be this:
try {
Connect-ExchangeOnline #ConnectExchange
} catch {
Write-Verbose -Verbose ($_.Exception.Message)
$Wait = ($_.Exception.Message) | Select-String ('(?<=for )(.*)(?= seconds)') -AllMatches
$Count = ([int]$Wait.Matches.Value)
Start-Sleep -seconds $Count
Connect-ExchangeOnline #ConnectExchange
}

Related

PowerShell Remoting, Eventing, Runspace - sticky Event Subscriber

I am currently stuck at a problem involving a WMI Event subscriber on a remote session running in a background runspace.
Below is the representative code that reproduces the issue (the real code is too long to share here) but, essentially, through a PS script I remotely install advertised WSUS updates with reboots when necessary. It takes about 90 minutes end to end.
The issue I am trying to solve at the moment is that during the course of patching, the support staff inadvertently log in to the the server being remotely patched and do their admin activities. To remind them, I want to display a pop up message as soon as the user logs in when my remote script is running. I am trying to do it using a background runspace plugged into the main patching script. It makes use of WMI eventing on the target server (which is being patched) to monitor user logons and display message as soon as it detects one. The below code is working as I expect. It even survives target server reboots.
$RemoteServerName = 'Server1.contoso.com'
$UserLogonAlertScriptBlock = {
param ($SyncedHashTable)
$RemoteServerName = $SyncedHashTable.target
try {
$Session = New-PSSession -ComputerName $RemoteServerName -ErrorAction Stop
} catch {}
while($true){
if($Session.State -eq 'Opened') {
$RemoteMonitoringJob = Invoke-Command -Session $Session -AsJob -ScriptBlock {
$null = Register-WMIEvent -Query "SELECT * FROM __InstanceCreationEvent WITHIN 3 WHERE TargetInstance ISA 'Win32_LogonSession'" -SourceIdentifier 'User.Logon'
Wait-Event -SourceIdentifier "User.Logon" -Timeout 7200 | ForEach-Object {
msg * /TIME:7200 /V "User logon detected" | Out-Null
$_ | Remove-Event
}
}
while($RemoteMonitoringJob.State -in #('NotStarted','Running')) {
Start-Sleep -Seconds 1
}
} else {
while($true) {
try {
$Session = New-PSSession -ComputerName $RemoteServerName -ErrorAction Stop
} catch {}
if($Session.State -eq 'Opened') {
break
}
Start-Sleep -Seconds 1
}
}
}
}
$Runspace = [runspacefactory]::CreateRunspace()
$PowerShell = [powershell]::Create()
$PowerShell.runspace = $Runspace
$SyncedHashTable = [hashtable]::Synchronized(#{})
$SyncedHashTable.host = $host
$SyncedHashTable.target = $RemoteServerName
$Runspace.Open()
$handle = $PowerShell.AddScript($UserLogonAlertScriptBlock).AddArgument($SyncedHashTable).BeginInvoke()
Write-Host '$(Get-Date): Long running script execution targeting $RemoteServerName has started'
Start-Sleep -Seconds 120 # it usually runs for upto 90 minutes, with remote reboots of $RemoteServerName
Write-Host "$(Get-Date): The script execution has completed"
### The code that cleans up the sticky event subscriber on the target server needs to be added here
The part I am stuck at is after the script completes its execution. The wsmanprovhost.exe running on the target server continues to stick around and shows alert messages when new users log on. I think it's because of the WMI event listener still being active on the box, not releasing the remote PS session.
In the above code, I need help close that remote listener so wsmanprovhost.exe disappears.
Could you please help?
PS. I have referred to #mklement0 's response in the following post but still no joy: The RunSpace and its closure
Update:
I have managed to address the challenge by adding a Boolean flag into the SyncedHashtable which is passed to the background runspace. When I want to stop the remote logon monitoring, in the main script I flip the flag. Since it's inside a synced hashtable, I can monitor that inside the runspace and terminate the remote invoke command job in the run space. But I still had to forcibly kill the remote wsmprovhost.exe as it refuses to go. I could do it by getting the pid of the remote PS session in advance. Not the most elegant way to close a remote PS session but it does the job for me. It's just that since the remote session is continuously monitoring for user logon event, there does not appear to be a way to run a piece of code in that session to unsubscribe the WMI event source. Will do more testing to see if there is any side effect.

Passing active Powershell PSSession connection as argument in Start-Job

I am writing a script which gathers data from Exchange Online concerning mailbox permissions for each mailbox in our organization. To do this I gather all mailbox data into a variable then I use foreach to iterate through each mailbox and check the mailbox permissions applied to it. This takes time when you are working with over 15000 mailboxes.
I would like to use Powershell Jobs to speed this process up by having multiple jobs checking permissions and appending them to a single CSV file. Is there a way to pass an active PSSession into a new job so that the job "shares" the active session of the parent process that spawned the job and does not require a new one to be established?
I could place a New-PSSession call into the function but Microsoft has active session limits in Exchange Online PSSessions so it would limit the number of jobs I could have running at one time to 3. The rest would have to be queued through a while loop. If I can share a single session between multiple jobs I would be limited by computer resources rather than connection restrictions.
Has anyone successfully passed an active PSSession through to a job before?
Edit:
I've been working on using runspaces to try to accomplish this with Boe Prox's PoshRSJobs module. Still having some difficulty getting it to work properly. Doesn't create the CSV or append to it but only if I try to sort out the permissions within the foreach statement. The Write-Output inside the scriptblock only outputs the implicit remoting information too which is odd.
Code is below.
Connect-ToOffice365TenantPSSession
$mailboxes = Get-Mailbox -ResultSize 10 -IncludeInactiveMailbox
$indexCount = 1
foreach ($mailbox in $mailboxes) {
$script = #"
`$cred = Import-Clixml -Path 'C:\Users\Foo\.credentials\StoredLocalCreds.xml'
`$o365Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://outlook.office365.com/powershell-liveid/" -Credential `$cred -Authentication Basic -AllowRedirection
Import-PSSession `$o365Session -CommandName #('Get-Mailbox','Get-MailboxPermission')
`$internal_mailbox = `$Using:mailbox
`$mailboxPermissions = `$internal_mailbox | Get-MailboxPermission
foreach (`$permission in (`$mailboxPermissions | Where-Object {`$_.User -match 'tenantName|companyDomain'}))
{
`$userPermissions = `$permission | Select-Object Identity, User, AccessRights
`$permissionObject = [PSCustomObject]#{
"MailboxName" = `$userPermissions.Identity
"MailboxAddress" = `$internal_mailbox.PrimarySmtpAddress
"MailboxType" = `$internal_mailbox.RecipientTypeDetails
"UserWithAccess" = `$userPermissions.User
"AccessRights" = `$userPermissions.AccessRights
}
if (Test-Path 'C:\Scripts\MailboxPermissions.csv') {
`$permissionObject | Export-Csv 'C:\Scripts\MailboxPermissions.csv' -NoTypeInformation -Append
} else {
New-Item -Path 'C:\Scripts\MailboxPermissions.csv'
`$permissionObject | Export-Csv 'C:\Scripts\MailboxPermissions.csv' -NoTypeInformation -Append
}
Write-Output `$permissionObject
}
"#
$scriptBlock = [scriptblock]::Create($script)
$continue = $false
do
{
if ((Get-RSJob | Where-Object {$_.State -eq "Running"}).count -lt 3) {
Start-RSJob -Name "Mailbox $indexCount" -ScriptBlock $scriptBlock
$indexCount++
$continue = $true
}
else {
Start-Sleep 1
}
} while ($continue -eq $false)
}
Get-RSJob | Receive-RSJob
Thanks for the suggestions.
You have specifications here, but you are not showing code and yet, asking for an opinion.
That is, as many would say here, off topic, because the goal here is to assist with code that is not working or the like. So, at this point, have you tried what you are asking, and if so, what happened?
So, IMHO … Yet, since you are here. let's not send you away empty handed.
As per a similar ask here, the accepted answer delivered was...
… you have a current PSSession opened up on your console and then you
are attempting to use that same session in a background job. This will
not work because you are using two different PowerShell processes and
are unable to share the live data between the runspaces as it is
deserialized and loses all of its capabilities. You will need to look
at creating your own PowerShell runspaces if your goal is to share
live variables across multiple PowerShell sessions.
Even with the above, you still have those consumption limits you mention and technically based on your use case, you'd more than likely end up with serious performance issues as well.

Prorgess bar for `New-MailboxExportRequest`

I'm trying to create a script that can export a user's mailbox to a PST, remotely (Exchange Server 2010 console is installed on the server we're running this from, and the module is loaded correctly). It's being done using a script so our L2 admins do not have to manually perform the task. Here's the MWE.
$UserID = Read-Host "Enter username"
$PstDestination = "\\ExServer\Share\$UserID.pst"
$Date = Get-Date -Format "yyyyMMddhhmmss"
$ExportName = "$UserID" + "$Date"
try {
New-MailboxExportRequest -Mailbox $UserID -FilePath $PstDestination -Name $ExportName -ErrorAction Stop -WarningAction SilentlyContinue | Out-Null
# Loop through the process to track its status and write progress
do {
$Percentage = (Get-MailboxExportRequest -Name $ExportName | Get-MailboxExportRequestStatistics).PercentComplete
Write-Progress "Mailbox export is in progress." -Status "Export $Percentage% complete" -PercentComplete "$Percentage"
}
while ($Percentage -ne 100)
Write-Output "$UserID`'s mailbox has been successfully exported. The archive can be found at $PstDestination."
}
catch {
Write-Output "There was an error exporting the mailbox. The process was aborted."
}
The problem is, as soon as we initiate the export, the task gets Queued. Sometimes, the export remains queued for a very long time, and the script is currently unable to figure out when the task begins, and when it does, is unable to display the progress correctly. The export happens in the background, but the script remains stuck there. So anything after the export, does not get executed, and the whole thing then has to be done manually.
Please suggest a way to handle this?
I tried adding a wait timer and then a check to see if the export has begun. It didn't quite work as expected.
Two things. First one is more about performance/hammering Exchange with unnesacary requests in do/while loop. Start-Sleep -Seconds 1 (or any other delay that makes sense depending on the mailbox size(s)) inside the loop is a must.
Second: rather than wait for job to start, just resume it yourself:
if ($request.Status -eq 'Queued') {
$request | Resume-MailboxExportRequest
}

Is there a way to tell when a site is created and ready to use when creating it using powershell?

I'm creating a number of sites using a powershell script. Now, when each of the sites is finished, I want to activate features on it.
My problem is that when I do this, it takes some time before the site is ready. Especially in SharePoint Online it is hard to predict when the site is ready. I've tried using time-loops, but I was wondering if there is a status setting somewhere that I can query instead.
Any thoughts?
Actually we solved the problem. The siteCreationOperation has a property named isComplete. Iterate over this and pick up the boolean for further processing :)
https://msdn.microsoft.com/en-us/library/microsoft.online.sharepoint.tenantadministration.tenant.createsite(v=office.15).aspx
#Create the site using the properties
$tenant.CreateSite($properties) | Out-Null
...
...
$siteCreationOperation = $tenant.CreateSite($properties)
$ctx.Load($siteCreationOperation)
...
...
#Create the site in the tennancy
...
...
do
{
...
...
$ctx.Load($siteCreationOperation)
$ctx.ExecuteQuery()
Write-Host $siteCreationOperation.IsComplete
...
...
}
...
while (!$siteCreationOperation.IsComplete)
...
Here is something that worked for me:
Connect-SPOService -Url $adminUrl -Credential $credentials
while ((Get-SPOSite -Filter "Url -like '*$($properties.Url)*'").Status -ne "Active")
{
Write-Host "." -NoNewline
Sleep -s 10
}
Where admin URL is https://Contonso-Admin.sharepoint.com and the Properties.Url is the site I am looking for, so something like https://Contonso.sharepoint.com/sites/Test1
This will work but you are better testing for the site and then doing another loop to test for the last artifact that creates like a list or a library. I've only tested this on an on premise and not online instance
$site = Get-SPSite <Site here> -ErrorVariable err -ErrorAction SilentlyContinue -AssignmentCollection $assignmentCollection
if($err)
{
while($err)
{
Write-Host "Waiting for site to be created"
Start-Sleep -seconds 5
$site = Get-SPSite <site here> -ErrorVariable err -ErrorAction SilentlyContinue -AssignmentCollection $assignmentCollection
}
#while loop for the last artifact that you are waiting for as the site will create but it may not be fully ready
}
Cheers
Truez

Powershell Script Gives Error But Still Returns "Successfully"

I'm writing a program to create an AD account and enable an Exchange mailbox and I'm getting some strange behaviour from it.
First, although it creates an AD user successfully, the mailbox cannot be enabled because "MyPath/Mr. Example could not be found". I assume this is because of a time lag between the AD server and the Exchange server (the code is run on the Exchange server). While inserting a sleep period seems to fix this, is there any other possibility I may be missing?
Second, I get this error message in the "red error text" that Powershell uses. It seems that my try/catch statement is letting the error slip by, which causes my script to erroneously inform me that the task completed successfully. How could I force the error to be caught within the script rather than being displayed on the console? (I was running it dot-sourced during testing, which is how I noticed this.)
I've included the relevant parts below. Any help is appreciated! Thanks =)
Import-Module ActiveDirectory
Add-PSSnapin Microsoft.Exchange.Management.Powershell.Admin
$errorLog = #()
$masterLog = #()
$time = Get-Date
New-ADUser -SamAccountName "example" -Name "Mr. Example" -Path "DC=MyPath" -Enabled 1 -Server ADServer
try{
Enable-Mailbox -Identity "MyPath/Mr. Example" -Alias "example" -Database "DatabaseName"
}
catch{
$myError = $Error[0]
$errorLog = $errorLog + "[$time] Error enabling mailbox for $fullName`: $myError"
}
if($errorLog.length -eq 0){
echo "An Active Directory account and Exchange mailbox for Mr. Example has been successfully created!"
}
else{
foreach($event in $errorLog){
$masterLog = $masterLog + $event
}
}
There are terminating and non-terminating errors in PowerShell. Only the former are caught by try..catch. You can force errors to be terminating by setting
... -ErrorAction Stop
for a specific command, or
$ErrorActionPreference = Stop
for the entire script. See the Scripting Guy blog for more information.