I have a PowerShell script using Invoke-Command that has been in use for about a year that suddenly has been having very bad performance issues. What normally took 1 to 2 seconds to run now takes about 90 seconds. Confusingly, it doesn't always take a long time. In fact, I've been testing it many times throughout today and have seen it run perfectly fine with every attempt over a 10-20 minute period, and then it goes back to being abysmally slow for the next 20-40 minute test period.
A simple look at my test code:
$cred = New-Object System.Management.Automation.PSCredential $username, $securePassword
Write-Host "Running command..."
Invoke-Command -ComputerName $target -Credential $cred -ScriptBlock {
Write-Host "Hello"
}
Write-Host "Done"
The timing of the results goes something like this:
"Running command..."
65 seconds...
"Hello"
25 seconds...
"Done"
For the usage of this, I need to wait on the process and then be able to see the result of the command so I can't just throw -AsJob into the script and not wait for the output. What should I be looking for to find what's slowing this down? I've checked the target machine during a slow response and don't see unusual CPU or memory usage.
I think I finally found part of what's causing this... And the answer has to do with some details I didn't provide in my initial question.
The target machine for my script is a domain controller, and network is setup with two domain controllers (for fault tolerance). If I make calls to -ComputerName "my.domain.com", I occasionally see the long delays... But if I just use the machine's IP address instead of the domain name, it goes through immediately.
I still don't know why this only now just started having issues, and what the real root problem is... but this gives me something to at least have the script working until we can upgrade our environment.
Related
I am writing a powershell script to manipulate Exchange Online mailboxes.
I want this script to run with background jobs in parallel, so I'm trying to use PoshRSJobs (https://github.com/proxb/PoshRSJob) to create the jobs.
My code is:
Connect-ExchangeOnline -Credentials ...
Start-RSJob -ModulesToImport ExchangeOnlineManagement -Throttle $ProcesosConcurrentes -InputObject $jobs -ScriptBlock {
./migra_buzon.ps1 ...
}
Where:
$jobs is an arraylist where I have the parameter of the mailboxes I want to operate with
migra_buzon.ps1 is another powershell scripts that operates over one specified mailbox
The problem I have when I run this way is that in the jobs I have the error:
The term 'Add-MailboxPermission' is not recognized as a name of a cmdlet, function, script file, or executable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
Although other commands like Get-EXOMailbox are working correctly.
Looking for help I found that the problem can be related with the session, so I changed my code to:
Connect-ExchangeOnline -Credentials ...
Start-RSJob -ModulesToImport ExchangeOnlineManagement -Throttle $ProcesosConcurrentes -InputObject $jobs -ScriptBlock {
$o365session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://outlook.office365.com/powershell-liveid" -Credential $(Import-Clixml $Using:ExchangeCredentials) -Authentication "Basic" -AllowRedirection
Import-PSSession $o365Session -CommandName #('Add-MailboxPermission', 'Get-MailboxPermission')
./migra_buzon.ps1 ...
}
In this case, the problem I have is with the Exchange connection. After running a few jobs I'm getting the error:
[outlook.office365.com] Processing data from remote server outlook.office365.com failed with the following error message: Client did not get proper response from server. For more information, see the about_Remote_Troubleshooting Help topic.
Cannot validate argument on parameter 'Session'. The argument is null. Provide a valid value for the argument, and then try running the command again.
So my question is, what is the right way to run background jobs sharing the connection got in the main process?
Thanks
PS: I first tried to run jobs with Start-Job, but with this the problem is that each background job needs its own connection, so I got and maximum number of connections exceeded. And this is the reason I changed my code to Start-RSJob
It appears that you are hitting Exchange Online throttling limits.
If that indeed the case, you can try the following method.
How to relax PowerShell throttling
There is a relatively new customer facing way to increase or update PowerShell Throttling Policies.
Go to Microsoft 365 admin center.
Validate that you are logged in with the user that has the correct role assignment.
Click on the Need Help? Widget in the bottom right corner
Graphical user interface
Type Exchange PowerShell throttling in the search box and select “Temporarily update throttling policies for a migration”. Keep in mind that this is only applicable for 90 days. After 90 days, the throttles will return to back to the default values for that tenant.
MachSol offers Tenant management using a job engine, that allows you to do multiple operations using front-end and let the jobs handler take care of processing in background. You can give it a try:
https://www.machsol.com/machpanel-automation-for-microsoft-CSP-partners/
I see some questions about this topic, but I cannot get it working
Get-Service -Name Spooler -ComputerName (Get-Content c:\tmp\scripts\Servers\iservers.txt) |
Stop-Service -PassThru | Set-Service -StartupType Disabled -whatif
The code executes for each server on the txt file, and stops de service, but not disable the service.
Any help to get it work and/or Troubleshooting???
Regards.
How to approach this kind of problem
In automation, we work up to complexity, meaning you should start simply and then add on more features until you see where it breaks.
Right now, you're trying to do a bunch of operations in one single line:
Load a list of computers and
Reach out to the computers and Stop a service and
Also while doing this, set the service to not automatically start.
There are a lot of problems you can run into, like "what happens if these PCs aren't enabled for remoting", or "what if you need a different account to handle stopping or disabling a service".
When you're trying to figure it all out in one-line, you're in for a bad and frustrating time.
How to fix it
Start simply. Start with one computer that's nearby and definitely turned on.
Begin with reading a service. Can you even get this operation to run?
Get-Service -ComputerName SomePC123 Spooler
Status Name DisplayName
------ ---- -----------
Running spooler Print Spooler
If you run into an error, then first figure out how to be able to remote into that one PC and see if the Print Spooler is running. Then, you will know what steps to deploy to all of your machines to prepare them for remoting.
Then, once you can check if a service is running, you can add on the next step, try to stop the service.
So your code would start to look like this:
$computers = get-content .\someTextFile.txt
forEach($computer in $computers){
$service = Get-Service -ComputerName $computer Spooler
"status of spooler on $computer is $($service.Status), with start type of $($service.StartType)"
#todo, set start type to Disabled...
}
Eventually, you will have migrated each step out of the one-liner and you'll know where and why any given command is failing. This is the way.
Update - the original question claimed that I was able to successfully perform an Invoke-Command and then shortly after was unable to; I thought it was due to processes going on during login after a windows upgrade.
It turns out the PC was actually starting, running a quick batch/cmd file, and then restarting. This is what was leading to being able to do PS Remoting and then suddenly not. The restart was quick enough after first boot that I didn't realize it was happening. Sorry for the bad question.
For the curious, the machine was restarting because of a remnant of the Microsoft Deployment Toolkit in-place upgrade process. The way MDT completes its task-sequence post-upgrade is problematic for many reasons, and now I've got another to count.
Old details (no longer relevant, with incorrect assumption that machine was not restarting after first successful Invoke-Command):
I'm automating various things with VMs in Hyper-V using powershell and powershell remoting. I'll start up a VM and then want to run some commands on it via powershell.
I'm struggling with determining when I can safely start running the remote commands via things like Invoke-Command. I can't start immediately as I need to let the machine start up.
Right now I poll the VM with a one second sleep between calls until the following function returns $true:
function VMIsReady {
[CmdletBinding()]
Param(
[Parameter(Mandatory=$True)][object]$VM
)
$heartbeat = $vm.Heartbeat
Write-Host "vm heartbeat is $heartbeat"
if (($heartbeat -eq 'OkApplicationsHealthy') -or ($heartbeat -eq 'OkApplicationsUnknown'))
{
try
{
Invoke-Command -VMName $vm.Name -Credential $(GetVMCredentials) {$env:computername} | out-null
}
catch [System.Management.Automation.RuntimeException]
{
Write-Host 'Caught expected automation runtime exception'
return $false
}
Write-Host 'remoting ready'
return $true
}
}
This usually works well; however, after a windows upgrade has happened, there are issues. I'll get Hyper-V remoting errors of various sorts even after VMIsReady returns $true.
These errors are happening while the VM is in the process of first user login after upgrade (Windows going through "Hi;We've got some updates for your PC;This might take several minutes-Don't turn off your PC). VMIsReady returns true right as this sequence starts - I imagine I probably should be waiting until the sequence is done, but I've no idea how to know when that is.
Is there a better way of determining when the machine is in a state where I can expect remoting to work without issue? Perhaps a way to tell when a user is fully logged on?
You can use Test-WSMan.
Of run a script on the invoke that will receive a response from the server.
[bool]$Response | Out-Null
try{
$Response = Invoke-Command -ComputerName Test-Computer -ScriptBlock {return $true}
}catch{
return $false
}
if ($Response -ne $true){
return $false
}else{
return $true
}
I have the following script to launch an application:
add-type -AssemblyName microsoft.VisualBasic
add-type -AssemblyName System.Windows.Forms
$args = "arguments"
$proc = Start-Process -PassThru "path" -ArgumentList $args
start-sleep -Seconds 5
[Microsoft.VisualBasic.Interaction]::AppActivate($proc.Id)
[System.Windows.Forms.SendKeys]::SendWait("~")
I use this script to launch several windows forms applications that need user interaction to start (i.e. push a start button). So far I've been sleeping the script 5 seconds to allow the application to launch. I tried to run this script in a variety of computers with different processing capabilities, but did not work in all computers, some launch the application slower than others, when the app took more than 5 seconds to launch the app the script fails, because the AppActivate could not find the PID.
I don't want to try different sleeping times for each computer, because I have to run this script in more than 100 computers at boot time.
I would like to know if there is a way to wait for the application to launch in a event driven way.
UPDATE:
The WaitForInputIdle does not work in all applications I tried to start. It returns immediately (successfully because it's returning true) and the AppActive method throws an exception.
I suggest you to avoid using "Sleep" to synchronize system objects (it's never a good solution). As far as you are starting Windows Forms application I suggest you to use Process.WaitForInputIdle Method.
$p = [diagnostics.process]::start("notepad.exe", "D:\temp\1mbfile.txt")
$p.WaitForInputIdle(5000);
using it in a loop you can test alternatively test if the input is idle (application is started and waiting) or if the application is stopped ($p.HasExited -eq $true).
do
{
if ($p.HasExited -eq $true)
{
break
}
} while ($p.WaitForInputIdle(5000) -ne $true)
So, I finally wrote this workaround since WaitForInputIdle does not work with all my applications and I was not able to figure out why.
do{
if ($proc.MainWindowHandle -ne 0)
{
break
}
$proc.Refresh()
} while ($true)
I am not sure if this is the best approach; anyway, I wanted to share this.
Neither of the other answers waited long enough for my script, so I came up with this, which waits until the process stops using the cpu.
do{
$ProcCpu = $Process.CPU
Start-Sleep -Milliseconds 100
} until($ProcCpu -eq $Process.CPU)
Caveat: Without spiking the cpu while a Get-WmiObject call parses the whole event log to match my specified filter.
Situation: I am working on a script that remotely runs some checks, then reboots a pc. I want it to check the health once the server reboots (after sleeping for some time) to make sure services that were supposed to start did. I've been running into "Automatic" services that start and then shut down (as intended) but then my current version picks them up as failed if they've already run. It was suggested that I check the event log for "Service Control Manager" errors, and report on those, the only problem now is that with the below script, we have servers who's event log can range anywhere from 20K to several hundred thousand events, and on a 2k server with 20K, this takes roughly 20 seconds to complete, and the cpu pegs near 100% while it's running.
I'm still learning powershell/wmi, so any advice would be appreciated.
function Check_Startup_Events {
BEGIN {
$time = [System.Management.ManagementDateTimeConverter]::ToDmtfDateTime((Get-Date).AddMinutes(-15))
}
PROCESS {
$results = Get-WmiObject Win32_NTLogEvent -computername $_ -Filter "LogFile='System' and SourceName='Service Control Manager' and TimeGenerated>='$time' and EventType=1" |
Format-Table -Autosize EventCode, Message
$results
}
}
$results = Get-EventLog -ComputerName w2kserver -LogName System -After $time
foreach ($result in $results){
if ($result.Source -eq "Service Control Manager" -and $result.EntryType -eq "Error"){
Write-Host $_.Description}}
I ran this against a 60k big event log on a W2K server in our environment. It takes a while to run but runs locally and does not tax the server. Not sure how you would want to output the data but I think Get-EventLog will do what you want.