Here is my Powershell code that when i execute it, it will continually loop while waking up my internal sites
$urlFile = "C:\Users\lfouche\Desktop\url\url.txt"
foreach($site in Get-Content $urlFile){
function WakeUp([string] $url)
{
Write-Host "Waking up $url ..." -NoNewLine
$client = new-object system.net.WebClient
$client.UseDefaultCredentials = $true
$null = $client.OpenRead($url)
$client.Dispose()
Write-Host " Ok"
}
#(
Get-Content $urlFile
) | % { WakeUp $_ }
}
Well, your script can generally be improved, however I think the error is that you already iterate over Get-Content $urlFile within your foreach condition and also in the loop. Try this:
# define the wake up function once
function WakeUp
{
Param
(
[string] $url
)
Write-Host "Waking up $url ..." -NoNewLine
$client = new-object system.net.WebClient
$client.UseDefaultCredentials = $true
$null = $client.OpenRead($url)
$client.Dispose()
Write-Host " Ok"
}
$urlFile = "C:\Users\lfouche\Desktop\url\url.txt"
$sites = Get-Content $urlFile
foreach($site in $sites)
{
# call wake up for each site
WakeUp $site
}
Related
Trying to use PowerShell to capture the running status of the "Nessus Essentials" software product. Simply trying to capture product status: running, not running, or other. Getting the below error each time. I've tried changing -like to -match and changing string [warn] [scanner] Not linked to a manager to various other shorter versions, with wildcards and without, to no avail. I still get several lines of an ugly error message when all I want is one line with the string Not linked to a manager returned to console with nothing beneath that.
Pertinent snippet working incorrectly:
} elseif(($agentStatus.stdOut -like "[warn] [scanner] Not linked to a manager")) {
Throw "Not linked to a manager"
The Error:
The Code:
Function Start-ProcessGetStreams {
[CmdLetBinding()]
Param(
[System.IO.FileInfo]$FilePath,
[string[]]$ArgumentList
)
$pInfo = New-Object System.Diagnostics.ProcessStartInfo
$pInfo.FileName = $FilePath
$pInfo.Arguments = $ArgumentList
$pInfo.RedirectStandardError = $true
$pInfo.RedirectStandardOutput = $true
$pinfo.UseShellExecute = $false
$pInfo.CreateNoWindow = $true
$pInfo.WindowStyle = [System.Diagnostics.ProcessWindowStyle]::Hidden
$proc = New-Object System.Diagnostics.Process
$proc.StartInfo = $pInfo
Write-Verbose "Starting $FilePath"
$proc.Start() | Out-Null
Write-Verbose "Waiting for $($FilePath.BaseName) to complete"
$proc.WaitForExit()
$stdOut = $proc.StandardOutput.ReadToEnd()
$stdErr = $proc.StandardError.ReadToEnd()
$exitCode = $proc.ExitCode
Write-Verbose "Standard Output: $stdOut"
Write-Verbose "Standard Error: $stdErr"
Write-Verbose "Exit Code: $exitCode"
[PSCustomObject]#{
"StdOut" = $stdOut
"Stderr" = $stdErr
"ExitCode" = $exitCode
}
}
Function Get-NessusStatsFromStdOut {
Param(
[string]$stdOut
)
$stats = New-Object System.Collections.Hashtable
$StdOut -split "`r`n" | % {
if($_ -like "*:*") {
$result = $_ -split ":"
$stats.add(($result[0].Trim() -replace "[^A-Za-z0-9]","_").ToLower(),$result[1].Trim())
}
}
Return $stats
}
Function Get-DateFromEpochSeconds {
Param(
[int]$seconds
)
$utcTime = (Get-Date 01.01.1970)+([System.TimeSpan]::fromseconds($seconds))
Return Get-Date $utcTime.ToLocalTime() -Format "yyyy-MM-dd HH:mm:ss"
}
Try {
$nessusExe = Join-Path $env:ProgramFiles -ChildPath "Tenable\Nessus\nessuscli.exe" -ErrorAction Stop
} Catch {
Throw "Cannot find NessusCli.exe"
}
Write-Host "Getting Agent Status..."
$agentStatus = Start-ProcessGetStreams -FilePath $nessusExe -ArgumentList "managed status"
If($agentStatus.stdOut -eq "" -and $agentStatus.StdErr -eq "") {
Throw "No Data Returned from NessusCli"
} elseif($agentStatus.StdOut -eq "" -and $agentStatus.StdErr -ne "") {
Throw "StdErr: $($agentStatus.StdErr)"
} elseif(($agentStatus.stdOut -like "[warn] [scanner] Not linked to a manager")) {
Throw "Not linked to a manager"
} elseif(-not($agentStatus.stdOut -like "*Running: *")) {
Throw "StdOut: $($agentStatus.StdOut)"
} else {
$stats = Get-NessusStatsFromStdOut -stdOut $agentStatus.StdOut
If($stats.last_connection_attempt -as [int]) { $stats.last_connection_attempt = Get-DateFromEpochSeconds $stats.last_connection_attempt }
If($stats.last_connect -as [int]) { $stats.last_connect = Get-DateFromEpochSeconds $stats.last_connect }
If($stats.last_scanned -as [int]) { $stats.last_connect = Get-DateFromEpochSeconds $stats.last_scanned }
}
$stats | Out-Host
Note: Code above is courtesy of here, I've only made a change to the path of Nessus, and I am adding the attempt to capture that it's not connected to a manager.
Modify your code so that it separates standard output from error and so that it handles each line separately.
The following is how to capture standard output (excluding else statements and error handling) of a program (according to your $Proc variable)
if ($proc.Start())
{
while (!$proc.StandardOutput.EndOfStream)
{
$StreamLine = $proc.StandardOutput.ReadLine()
if (![string]::IsNullOrEmpty($StreamLine))
{
# TODO: Duplicate code in this scope as needed or rewrite to use multiline regex
$WantedLine = [regex]::Match($StreamLine, "(?<wanted>.*Not linked to a manager.*)")
$Capture = $WantedLine.Groups["wanted"]
if ($Capture.Success)
{
Write-Output $Capture.Value
}
}
}
}
After that deal with error output separately:
$StandardError = $Proc.StandardError.ReadToEnd()
if (![string]::IsNullOrEmpty($StandardError))
{
# Or use Write-Error
Write-Output $StandardError
}
I need to download some webcontent from many servers in parallel as part of a scheduled job, but I cannot find a correct way to run the download in parallel/async. How can this be done?
Without any parallelism I can do it this way, but it is very slow:
$web = [System.Net.WebClient]::new()
[Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
# $srvList is a list of servers of viariable length
$allData = ""
foreach ($srv in $srvList) {
$url = "https:\\$srv\MyWebPage"
$data = $web.DownloadString($url)
$allData += $data
}
But how to do this in parallel via "$web.DownloadStringAsync"?
I found this snippet, but I dont see how to get the result of each call and how to concatenate it:
$job = Register-ObjectEvent -InputObject $web -EventName DownloadStringCompleted -Action {
Write-Host 'Download completed'
write-host $EventArgs.Result
}
$web.DownloadString($url)
Does someone know, how to get this solved in a short & smart way?
The best and fastest way is using runspaces:
Add-Type -AssemblyName System.Collections
$GH = [hashtable]::Synchronized(#{})
[System.Collections.Generic.List[PSObject]]$GH.results = [System.Collections.Generic.List[string]]::new()
[System.Collections.Generic.List[string]]$GH.servers = #('server1','server2');
[System.Collections.Generic.List[string]]$GH.functions = #('Download-Content');
[System.Collections.Generic.List[PSObject]]$jobs = #()
#-----------------------------------------------------------------
function Download-Content {
#-----------------------------------------------------------------
# a function which runs parallel
param(
[string]$server
)
$web = [System.Net.WebClient]::new()
[Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$url = "https:\\$server\MyWebPage"
$data = $web.DownloadString($url)
$GH.results.Add( $data )
}
#-----------------------------------------------------------------
function Create-InitialSessionState {
#-----------------------------------------------------------------
param(
[System.Collections.Generic.List[string]]$functionNameList
)
# Setting up an initial session state object
$initialSessionState = [System.Management.Automation.Runspaces.InitialSessionState]::CreateDefault()
foreach( $functionName in $functionNameList ) {
# Getting the function definition for the functions to add
$functionDefinition = Get-Content function:\$functionName
$functionEntry = New-Object System.Management.Automation.Runspaces.SessionStateFunctionEntry -ArgumentList $functionName, $functionDefinition
# And add it to the iss object
[void]$initialSessionState.Commands.Add($functionEntry)
}
return $initialSessionState
}
#-----------------------------------------------------------------
function Create-RunspacePool {
#-----------------------------------------------------------------
param(
[InitialSessionState]$initialSessionState
)
$runspacePool = [RunspaceFactory]::CreateRunspacePool(1, ([int]$env:NUMBER_OF_PROCESSORS + 1), $initialSessionState, $Host)
$runspacePool.ApartmentState = 'MTA'
$runspacePool.ThreadOptions = "ReuseThread"
[void]$runspacePool.Open()
return $runspacePool
}
#-----------------------------------------------------------------
function Release-Runspaces {
#-----------------------------------------------------------------
$runspaces = Get-Runspace | Where { $_.Id -gt 1 }
foreach( $runspace in $runspaces ) {
try{
[void]$runspace.Close()
[void]$runspace.Dispose()
}
catch {
}
}
}
$initialSessionState = Create-InitialSessionState -functionNameList $GH.functions
$runspacePool = Create-RunspacePool -initialSessionState $initialSessionState
foreach ($server in $GH.servers)
{
Write-Host $server
$job = [System.Management.Automation.PowerShell]::Create($initialSessionState)
$job.RunspacePool = $runspacePool
$scriptBlock = { param ( [hashtable]$GH, [string]$server ); Download-Content -server $server }
[void]$job.AddScript( $scriptBlock ).AddArgument( $GH ).AddArgument( $server )
$jobs += New-Object PSObject -Property #{
RunNum = $jobCounter++
JobObj = $job
Result = $job.BeginInvoke() }
do {
Sleep -Seconds 1
} while( $runspacePool.GetAvailableRunspaces() -lt 1 )
}
Do {
Sleep -Seconds 1
} While( $jobs.Result.IsCompleted -contains $false)
$GH.results
Release-Runspaces | Out-Null
[void]$runspacePool.Close()
[void]$runspacePool.Dispose()
Finally I found a simple solution via events. Here is my code-snippet:
cls
Remove-Variable * -ea 0
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = $null
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$srvList = #('srv1','srv2','srv3')
$webObjList = [System.Collections.ArrayList]::new()
$eventList = [System.Collections.ArrayList]::new()
$resultList = [System.Collections.ArrayList]::new()
$i=0
foreach ($srv in $srvList) {
$null = $webObjList.add([System.Net.WebClient]::new())
$null = $eventList.add($(Register-ObjectEvent -InputObject $webObjList[$i] -EventName DownloadStringCompleted -SourceIdentifier $srv))
$null = $resultList.add($webObjList[$i].DownloadStringTaskAsync("https://$srv/MyWebPage"))
$i++
}
do {sleep -Milliseconds 10} until ($resultList.IsCompleted -notcontains $false)
foreach ($srv in $srvList) {Unregister-Event $srv}
# show all Results:
$resultList.result
I would like to ask you, how it is possible to handle multiple connection threads.
I have implemented TCP server in the following way:
$endpoint = New-Object System.Net.IPEndPoint ([System.Net.IPAddress]::Any, 8989)
$listener = New-Object System.Net.Sockets.TcpListener $endpoint
$listener.Start()
do {
$client = $listener.AcceptTcpClient() # will block here until connection
$stream = $client.GetStream();
$reader = New-Object System.IO.StreamReader $stream
do {
$line = $reader.ReadLine()
Write-Host $line -fore cyan
} while ($line -and $line -ne ([char]4))
$reader.Dispose()
$stream.Dispose()
$client.Dispose()
} while ($line -ne ([char]4))
$listener.Stop()
This code can handle just one thread in time.
Can you give me an advice on how to create a TCP server in PowerShell that can handle multiple clients?
For handling multiple clients you need multiple threads and for that you need to use runspaces. Below is the working code which accepts multiple clients and do the processing of each client in separate thread (runspace)
$Global:Listener = [HashTable]::Synchronized(#{})
$Global:CnQueue = [System.Collections.Queue]::Synchronized((New-Object System.collections.queue))
$Global:space = [RunSpaceFactory]::CreateRunspace()
$space.Open()
$space.SessionStateProxy.setVariable("CnQueue", $CnQueue)
$space.SessionStateProxy.setVariable("Listener", $Listener)
$Global:newPowerShell = [PowerShell]::Create()
$newPowerShell.Runspace = $space
$Timer = New-Object Timers.Timer
$Timer.Enabled = $true
$Timer.Interval = 1000
Register-ObjectEvent -SourceIdentifier MonitorClientConnection -InputObject $Timer -EventName Elapsed -Action {
While($CnQueue.count -ne 0) {
$client = $CnQueue.Dequeue()
$newRunspace = [RunSpaceFactory]::CreateRunspace()
$newRunspace.Open()
$newRunspace.SessionStateProxy.setVariable("client", $client)
$newPowerShell = [PowerShell]::Create()
$newPowerShell.Runspace = $newRunspace
$process = {
$stream = $client.GetStream();
$reader = New-Object System.IO.StreamReader $stream
[console]::WriteLine("Inside Processing")
# You have client here so do whatever you want to do here.
# This is a separate thread so if you write blocking code here, it will not impact any other part of the program
}
$jobHandle = $newPowerShell.AddScript($process).BeginInvoke()
#jobHandle you need to save for future to cleanup
}
}
$listener = {
$Listener['listener'] = New-Object System.Net.Sockets.TcpListener("127.0.0.1", "1234")
$Listener['listener'].Start()
[console]::WriteLine("Listening on :1234")
while ($true) {
$c = $Listener['listener'].AcceptTcpClient()
If($c -ne $Null) {
[console]::WriteLine("{0} >> Accepted Client " -f (Get - Date).ToString())
$CnQueue.Enqueue($c)
}
Else {
[console]::WriteLine("Shutting down")
Break
}
}
}
$Timer.Start()
$Global:handle = $newPowerShell.AddScript($listener).BeginInvoke()
For more detailed example please go here
I'm writing a script that will watch a directory for any new mp4 files, then convert the file using HandBrake's CLI tool. The logic that watches the directory for changes works by itself, but if I drop a large video into the "watched" directory the conversion fails since it kicks off as soon as it sees a new file, before the file has time to finish copying.
I'm using a do until loop to check if the file is locked/downloading and then continues once the file is unlocked/writable. The loop works as a stand-alone script, but when used inside the filesystem watcher the script it will halt without any errors on this line:
[System.IO.FileStream] $fs = $convertingFile.OpenWrite();
This occurs regardless if $ErrorActionPreference = "SilentlyContinue" is commented out. I've been unable to gather any log output to see why the script is halting using Start-Transcript or Out-File.
How should I best gather error info about why the script halts once it hits this line?
Bonus: Why might this script not provide error information?
$ErrorActionPreference = "SilentlyContinue"
function Start-FileSystemWatcher {
[CmdletBinding()]
param(
[Parameter()]
[string]$Path,
[Parameter()]
[ValidateSet('Changed','Created','Deleted','Renamed')]
[string[]]$EventName,
[Parameter()]
[string]$Filter,
[Parameter()]
[System.IO.NotifyFilters]$NotifyFilter,
[Parameter()]
[switch]$Recurse,
[Parameter()]
[scriptblock]$Action
)
#region Build FileSystemWatcher
$FileSystemWatcher = New-Object System.IO.FileSystemWatcher
if (-not $PSBoundParameters.ContainsKey('Path')) {
$Path = $PWD
}
$FileSystemWatcher.Path = $Path
if ($PSBoundParameters.ContainsKey('Filter')) {
$FileSystemWatcher.Filter = $Filter
}
if ($PSBoundParameters.ContainsKey('NotifyFilter')) {
$FileSystemWatcher.NotifyFilter = $NotifyFilter
}
if ($PSBoundParameters.ContainsKey('Recurse')) {
$FileSystemWatcher.IncludeSubdirectories = $True
}
if (-not $PSBoundParameters.ContainsKey('EventName')) {
$EventName = 'Changed','Created','Deleted','Renamed'
}
if (-not $PSBoundParameters.ContainsKey('Action')) {
$Action = {
switch ($Event.SourceEventArgs.ChangeType) {
'Renamed' {
$Object = "{0} was {1} to {2} at {3}" -f $Event.SourceArgs[-1].OldFullPath,
$Event.SourceEventArgs.ChangeType,
$Event.SourceArgs[-1].FullPath,
$Event.TimeGenerated
}
Default {
$Object = "{0} was {1} at {2}" -f $Event.SourceEventArgs.FullPath,
$Event.SourceEventArgs.ChangeType,
$Event.TimeGenerated
}
}
$WriteHostParams = #{
ForegroundColor = 'Green'
BackgroundColor = 'Black'
Object = $Object
}
Write-Host #WriteHostParams
}
}
$ObjectEventParams = #{
InputObject = $FileSystemWatcher
Action = $Action
}
foreach ($Item in $EventName) {
$ObjectEventParams.EventName = $Item
$ObjectEventParams.SourceIdentifier = "File.$($Item)"
Write-Verbose "Starting watcher for Event: $($Item)"
$Null = Register-ObjectEvent #ObjectEventParams
}
}
$FileSystemWatcherParams = #{
Path = 'X:\share\scripts\ps\converter\input'
Recurse = $True
NotifyFilter = 'FileName'
Verbose = $True
Action = {
$Item = Get-Item $Event.SourceEventArgs.FullPath
$WriteHostParams = #{
ForegroundColor = 'Green'
BackgroundColor = 'Black'
}
$inputFile = "${PWD}\input\$($Item.Name)".trim()
$outputFile = "${PWD}\output\$($Item.Name)".trim()
$logFile = "${PWD}\log\$($Item.Name).txt"
$testLogFile = "${PWD}\log\$($Item.Name)(t).txt"
function mp4-Func {
Start-Transcript -path $logFile
Write-Host "New mp4 file detected..."
$convertingFile = New-Object -TypeName System.IO.FileInfo -ArgumentList $inputFile
$locked = 1
do {
[System.IO.FileStream] $fs = $convertingFile.OpenWrite();
if (!$?) {
Write-Host "Can't convert yet, file appears to be loading..."
sleep 2
}
else {
$fs.Dispose()
$locked = 0
}
} until ($locked -eq 0)
Write-Host "File unlocked and ready for conversion."
HandBrake
Stop-Transcript
$WriteHostParams.Object = "Finished converting: $($Item.Name)"
}
function HandBrake {
.\HandBrakeCLI.exe --input "$inputFile" `
--output "$outputFile" `
--format av_mp4 `
--encoder x264 `
--vb 1700 `
--two-pass `
--aencoder copy:aac `
--ab 320 `
--arate 48 `
--mixdown stereo
}
switch -regex ($Item.Extension) {
'\.mp4' { mp4-Func }
}
Write-Host #WriteHostParams
}
}
#( 'Created') | ForEach-Object {
$FileSystemWatcherParams.EventName = $_
Start-FileSystemWatcher #FileSystemWatcherParams
}
I think you'll find that $ErrorActionPreference only impact errors at the level of cmdlets, while you hitting a problem in non-cmdlet code. For that, you'll probably need a try/catch construct.
Based on Burt_Harris' answer (please upvote his answer over this one), I changed the "do while" loop so that it would use try/catch rather than an if/else statement. By using $.Exception.Message and $.Exception.ItemName I was able to better understand why the script was halting at that particular line.
Working code:
Do {
Try {
[System.IO.FileStream] $fs = $convertingFile.OpenWrite()
$fs.Dispose()
$locked = 0
}
Catch {
$ErrorMessage = $_.Exception.Message
$FailedItem = $_.Exception.ItemName
Write-Host $ErrorMessage
Write-Host $FailedItem
Write-Host "Can't convert yet, file appears to be loading..."
sleep 5
continue
}
} until ($locked -eq 0)
#http://stackoverflow.com/a/24036900/175063
$user = "uuuu"
$pwd = "pppp"
$source = "http://1.1.1.1/manager/jmxproxy?get=java.lang:type=Memory&att=HeapMemoryUsage"
$destination = "D:\Work\ps\test.xml"
$wc = new-object System.Net.WebClient
$p = New-Object System.Net.WebProxy 'http://proxy:8080'
$p.UseDefaultCredentials = $true
$wc.proxy = $p
$credCache = New-Object System.Net.CredentialCache
$creds = New-Object System.Net.NetworkCredential($user, $pwd)
$credCache.Add($source, "Basic", $creds)
$wc.Credentials = $credCache
$wc.DownloadFile($source, $destination)
# max=1445462016, used=898674904
# free
foreach ($thing in Get-Content $destination) {
$max = $thing.split("max=")
$used = $thing.split("used=")
Write-Host $max
Write-Host $used
}
#$free = $max - $used
#Write-Host $free
The string the file that is downloaded is a one-liner:
OK - Attribute get 'java.lang:type=Memory' - HeapMemoryUsage= javax.management.openmbean.CompositeDataSupport(compositeType=javax.management.openmbean.CompositeType(name=java.lang.management.MemoryUsage,items=((itemName=committed,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=init,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=max,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)),(itemName=used,itemType=javax.management.openmbean.SimpleType(name=java.lang.Long)))),contents={committed=1444478976, init=1494220800, max=1445462016, used=868228272})
And all I really want from it is:
max=1445462016
used=868228272
to be:
1445462016-868228272=577233744
I would extract the values from the contents={...} portion of the string with a regular expression, replace the commas with newlines and convert the result to a hashtable. Then you just need to cast the values to integers for the calculation.
Get-Content $destination | Where-Object {
$_ -match ',contents=\{(.+?)\}'
} | ForEach-Object {
$values = $matches[1] -replace ', ', "`n" | ConvertFrom-StringData
$free = [int]$values['max'] - [int]$values['used']
'Max: {0}' -f $values['max']
'Used: {0}' -f $values['used']
'Free: {0}' -f $free
}
I think I interpreted this correctly, do you want to display this in the console or store it in a variable, I'm assuming console here:
write-host "$max-$used="($max-$used)
I have figured out my own solution... I know it may not be the best way, but seems to work..
#http://stackoverflow.com/a/24036900/175063
$user="uuuu"
$pwd="pppp"
$source="http://1.1.1.1/manager/jmxproxy?get=java.lang:type=Memory&att=HeapMemoryUsage"
$destination="D:\Work\ps\test.xml"
$wc=new-object System.Net.WebClient
$p = New-Object System.Net.WebProxy 'http://proxy:8080'
$p.UseDefaultCredentials = $true
$wc.proxy=$p
$credCache=new-object System.Net.CredentialCache
$creds=new-object System.Net.NetworkCredential($user,$pwd)
$credCache.Add($source, "Basic", $creds)
$wc.Credentials=$credCache
$wc.DownloadFile($source, $destination)
# max=1445462016, used=898674904
# free
foreach ($thing in Get-Content $destination) {
# , max=1445462016, used=696318832})
# $a = $a.substring(2,3)
# MID: https://technet.microsoft.com/en-us/library/ee176901.aspx
# LEN: https://technet.microsoft.com/en-us/library/ee176895.aspx
# Instr: https://technet.microsoft.com/en-us/library/ee176876.aspx
$len = $thing.length
$maxst = $thing.indexof("max=")
$usedst = $thing.indexof("used=")
$max=$thing.substring($maxst+4,$len-$usedst-6)
$used=$thing.substring($usedst+5,$len-$usedst-7)
$free=$max-$used
# , max=1445462016, used=696318832})
write-host $len
write-host $maxst
write-host $usedst
write-host $max
write-host $used
write-host $free
}