I have a powershell script with a while loop, when itr first start the script its using 15,504kb memory, but it looks like every time it goes through the while loop the memory usage is increasing by about 100kb, and it never seems to go down.
$exe = 'C:\Program Files (x86)\MyApp\MyApp.exe'
$logOutput = 'C:\Log\Log.log'
$logdir1 = 'C:\Log'
$storage = 'ext'
$date = (Get-Date)
"{0:dd/MM/yy HH:mm:ss} - START" -f $date | Out-File -filepath $logOutput -NoClobber -Append -Encoding ASCII
While ($true){
$date = (Get-Date)
"{0:dd/MM/yy HH:mm:ss} - Calling exe for D:\" -f $date | Out-File -filepath $logOutput -NoClobber -Append -Encoding ASCII
& $exe /logdir $logdir1 /storage $storage
Start-Sleep -s 5
$date = (Get-Date)
"{0:dd/MM/yy HH:mm:ss} - Calling exe for E:\" -f $date | Out-File -filepath $logOutput -NoClobber -Append -Encoding ASCII
& $exe /logdir $logdir1 /storage $storage
Start-Sleep -s 10
}
Is there something wrong with my script?
Related
I am a programming enthusiast and novice, I am using Powershell to try to solve the following need:
I need to extract the full path of files with extension .img. inside a folder with +/- 900 thousand folders and +/- million files. -/+ 900,000 img files.
Each img file must be processed in an exe. that is read from a file.
Which is better to store the result of the GetChildItem in a variable or a file?
I would greatly appreciate your guidance and support to optimize and / or find the best way to speed up processes vs. resource consumption.
Thank you un advance!!
This is the code I am currently using:
$PSDefaultParameterValues['*:Encoding'] = 'Ascii'
$host.ui.RawUI.WindowTitle = “DICOM IMPORT IN PROGRESS”
#region SET WINDOW FIXED WIDTH
$pshost = get-host
$pswindow = $pshost.ui.rawui
$newsize = $pswindow.buffersize
$newsize.height = 3000
$newsize.width = 150
$pswindow.buffersize = $newsize
$newsize = $pswindow.windowsize
$newsize.height = 50
$newsize.width = 150
$pswindow.windowsize = $newsize
#endregion
#
$out = ("$pwd\log_{0:yyyyMMdd_HH.mm.ss}_import.txt" -f (Get-Date))
cls
"`n" | tee -FilePath $out -Append
"*****************" | tee -FilePath $out -Append
"**IMPORT SCRIPT**" | tee -FilePath $out -Append
"*****************" | tee -FilePath $out -Append
"`n" | tee -FilePath $out -Append
#
# SET SEARCH FOLDERS #
"Working Folder" | tee -FilePath $out -Append
$path1 = Read-Host "Enter folder location" | tee -FilePath $out -Append
"`n" | tee -FilePath $out -Append
#
#
# SET & SHOW HOSTNAME
"SERVER NAME" | tee -FilePath $out -Append
$ht = hostname | tee -FilePath $out -Append
Write-Host $ht
Start-Sleep -Seconds 3
"`n" | tee -FilePath $out -Append
#
#
# GET FILES
"`n" | tee -FilePath $out -Append
#"SEARCHING IMG FILES, PLEASE WAIT..." | tee -FilePath $out -Append
$files = $path1 | Get-ChildItem -recurse -file -filter *.img | ForEach-Object { $_.FullName }
# SHOW Get-ChildItem PROCESS ON CONSOLE
Out-host -InputObject $files
"`n" | tee -FilePath $out -Append
Write-Output ($files | Measure).Count "IMG FILES FOUND TO PUSH" | tee -FilePath $out -Append
# DUMP Get-ChildIte into a file
$files > $pwd\pf
Start-Sleep -Seconds 5
# TIMESTAMP
"`n" | tee -FilePath $out -Append
"IMPORT START" | tee -FilePath $out -Append
("{0:yyyy/MM/dd HH:mm:ss}" -f (Get-Date)) | tee -FilePath $out -Append
"********************************" | tee -FilePath $out -Append
"`n" | tee -FilePath $out -Append
#
#
#SET TOOL
$ir = $Env:folder_tool
$pt = "utils\tool.exe"
#
#PROCESSING FILES
$n = 1
$pe = foreach ($file in Get-Content $pwd\pf ) {
$tb = (Get-Date -f HH:mm:ss) | tee -FilePath $out -Append
$fp = "$n. $file" | tee -FilePath $out -Append
#
$ep = & $ir$pt -c $ht"FIR" -i $file | tee -FilePath $out -Append
$as = "`n" | tee -FilePath $out -Append
# PRINT CONSOLE IMG FILES PROCESS
Write-Host $tb
Write-Host $fp
Out-host -InputObject $ep
Write-Host $as
$n++
}
#
#TIMESTAMP
"********************************" | tee -FilePath $out -Append
"IMPORT END" | tee -FilePath $out -Append
("{0:yyyy/MM/dd HH:mm:ss}" -f (Get-Date)) | tee -FilePath $out -Append
"`n" | tee -FilePath $out -Append
Try using parallel with PoshRSJob.
Replace Start-Process in Process-File with your code and note that there is no access to console. Process-File must return string.
Adjust $JobCount and $inData.
The main idea is to load all file list into ConcurrentQueue, start 20 background jobs and wait them to exit. Each job will take value from queue and pass to Process-File, then repeat until queue is empty.
NOTE: If you stop script, RS Jobs will continue to run until they finished or powershell closed. Use Get-RSJob | Stop-RSJob and Get-RSJob | Remove-RSJob to stop background work
Import-Module PoshRSJob
Function Process-File
{
Param(
[String]$FilePath
)
$process = Start-Process -FilePath 'ping.exe' -ArgumentList '-n 5 127.0.0.1' -PassThru -WindowStyle Hidden
$process.WaitForExit();
return "Processed $FilePath"
}
$JobCount = [Environment]::ProcessorCount - 2
$inData = [System.Collections.Concurrent.ConcurrentQueue[string]]::new(
[System.IO.Directory]::EnumerateFiles('S:\SCRIPTS\FileTest', '*.img')
)
$JobScript = [scriptblock]{
$inQueue = [System.Collections.Concurrent.ConcurrentQueue[string]]$args[0]
$outBag = [System.Collections.Concurrent.ConcurrentBag[string]]$args[1]
$currentItem = $null
while($inQueue.TryDequeue([ref] $currentItem) -eq $true)
{
try
{
# Add result to OutBag
$result = Process-File -FilePath $currentItem -EA Stop
$outBag.Add( $result )
}
catch
{
# Catch error
Write-Output $_.Exception.ToString()
}
}
}
$resultData = [System.Collections.Concurrent.ConcurrentBag[string]]::new()
$i_cur = $inData.Count
$i_max = $i_cur
# Start jobs
$jobs = #(1..$JobCount) | % { Start-RSJob -ScriptBlock $JobScript -ArgumentList #($inData, $resultData) -FunctionsToImport #('Process-File') }
# Wait queue to empty
while($i_cur -gt 0)
{
Write-Progress -Activity 'Doing job' -Status "$($i_cur) left of $($i_max)" -PercentComplete (100 - ($i_cur / $i_max * 100))
Start-Sleep -Seconds 3 # Update frequency
$i_cur = $inData.Count
}
# Wait jobs to complete
$logs = $jobs | % { Wait-RSJob -Job $_ } | % { Receive-RSJob -Job $_ }
$jobs | % { Remove-RSJob -Job $_ }
$Global:resultData = $resultData
$Global:logs = $logs
$Global:resultData is array of Process-File return strings
Which is better to store the result of the GetChildItem in a variable or a file?
If you're hoping to keep memory utilization low, the best solution is to not store them at all - simply consume the output from Get-ChildItem directly:
$pe = Get-ChildItem -Recurse -File -filter *.img |ForEach-Object {
$file = $_.FullName
$tb = (Get-Date -f HH:mm:ss) | tee -FilePath $out -Append
$fp = "$n. $file" | tee -FilePath $out -Append
#
$ep = & $ir$pt -c $ht"FIR" -i $file | tee -FilePath $out -Append
$as = "`n" | tee -FilePath $out -Append
# PRINT CONSOLE IMG FILES PROCESS
Write-Host $tb
Write-Host $fp
Out-host -InputObject $ep
Write-Host $as
$n++
}
I'm currently having connectivity issues and I would like to monitor a certain IP address and log the information.
I know I could use this command (using google.com as an example):
ping google.com /t |Foreach{"{0} - {1}" -f (Get-Date),$_} > pingORA.txt
In this case it will log a ping result every second. I would like it to log a ping result every 30 seconds. Is there a way to add this to the above command?
This can be done in PowerShell using Test-Connection.
$hostName = 'localhost'
$waitTime = 30
$logFile = 'pingORA.txt'
if (Test-Path -Path $logFile) { Remove-Item -Path $logFile }
while (1) {
$p = Test-Connection -ComputerName $hostName -Count 1 -ErrorAction SilentlyContinue
if ($null -ne $p) {
"{0} - {1} - {2}" -f (Get-Date), $hostName, $p.ProtocolAddress |
Out-File -FilePath $logFile -Encoding ascii -Append
} else {
"{0} - {1} - {2}" -f (Get-Date), $hostName, 'offline' |
Out-File -FilePath $logFile -Encoding ascii -Append
}
Start-Sleep -Seconds $waitTime
}
use awk to save every 30th ping result discarding the rest
awk '
BEGIN { srand();t=srand();\
while (1 ) {\
tElapsed=srand()-t;
if(tElapsed%30==0){\
print("write to file from here")\
}\
print ("pinging google");\
system("sleep 1");\
}
}'
Though it seems you rather wanted a powershell solution, (so I have included the relevant tag as well) You can still do the same in normal cmd by using a for loop:
for /f "delims=" %a in ('ping google.com -t') do echo %a >> pingORA.txt & timeout /t 30 >nul 2>&1
Note however, this is directly from cmd and not powershell
I have a powershell script which gathers some file information on some remote servers.
When writing to the console, I see the results just fine, but they don't seem to be getting written into the specified out-file:
out-file -filepath $filepath -inputobject $date -force -encoding ASCII -width 50
ForEach($server in $serverLists)
{
$result += $server
$result += $break
ForEach($filePath in $filePaths)
{
$result += $filePath
$result += $break
$command = "forfiles /P " + $filePath + " /S /D -1 > C:\temp\result.txt"
$cmd = "cmd /c $command"
Invoke-WmiMethod -class Win32_process -name Create -ArgumentList $cmd -ComputerName $server
sleep 2
$result += Get-Content \\$server\C$\temp\result.txt
$result += $break
$result += $break
write-host $result
#out-file -filepath $filepath -Append -inputobject $result -encoding ASCII -width 200
Remove-Item \\$server\C$\temp\result.txt
}
$result += $break
}
out-file -filepath $filepath -Append -inputobject $result -encoding ASCII -width 200
You use same variable $filepath in two places: as loop variable for foreach loop and outside of loop. foreach loop does not restore value of loop variable after loop ends. So, right after loop ends loop variable will have last value of loop or value where loop was interrupted by break command. If you does not want variable value to be overwritten by foreach loop, then you should choose different name for loop variable.
I have a PS script which Invoke-Expression on other scripts on the same computer.
Here is the code:
$webMemory = "C:\Memory_Script\WebMemory_Script.ps1"
$intMemory = "C:\Memory_Script\IntMemory_Script.ps1"
$hungWeb = "C:\Scripts\HungWeb_Script.ps1"
$hungInt = "C:\Scripts\HungInt_Script.ps1"
$intMemoryResult = #()
$webMemoryResult = #()
$hungWebResult = #()
$hungIntResult = #()
$date = Get-Date
$shortDate = (get-date -format ddMMyyy.hhmm)
$filepath = "C:\Scripts\Memory&HungResults\Results" + $shortdate + ".txt"
$break = "`r`n"
out-file -filepath $filepath -inputobject $date -force -encoding ASCII -width 50
out-file -filepath $filepath -Append -inputobject $break -encoding ASCII -width 50
$intMemoryResult += Invoke-Expression $intMemory
$webMemoryResult += Invoke-Expression $webMemory
$hungWebResult += Invoke-Expression $hungWeb
$hungIntResult += Invoke-Expression $hungInt
Write-host $webMemoryResult
out-file -filepath -Append -inputobject $intMemoryResult -encoding ASCII -width 200
out-file -filepath -Append -inputobject $break -encoding ASCII -width 200
out-file -filepath -Append -inputobject $webMemoryResult -encoding ASCII -width 200
out-file -filepath -Append -inputobject $break -encoding ASCII -width 200
out-file -filepath -Append -inputobject $hungIntResult -encoding ASCII -width 200
out-file -filepath -Append -inputobject $break -encoding ASCII -width 200
out-file -filepath -Append -inputobject $hungWebResult -encoding ASCII -width 200
out-file -filepath -Append -inputobject $break -encoding ASCII -width 200
Code in one of the scripts being called (the other three have similar functions)
$serverList = #("List of servers")
$w3wpMemory = #()
$w3wpMemory += "---------- W3WP Memory Consumption ----------"
$w3wpresult = #()
$toBeRecycled =#()
$toBeRecycled += "******************** THE INT SERVERS BELOW NEED TO BE RECYCLED (Hung) ********************" + "`r`n"
$date = Get-Date
$shortDate = (get-date -format ddMMyyy.hhmm)
$filepath = "C:\Scripts\HungIntResults\HungServerResults" + $shortdate + ".txt"
$break = "`r`n"
out-file -filepath $filepath -inputobject $date -force -encoding ASCII -width 50
out-file -filepath $filepath -Append -inputobject $break -encoding ASCII -width 50
ForEach($server in $serverList)
{
$w3wpresult += (get-wmiobject Win32_Process -filter "commandline like '%serviceoptimization%'" -computername $server).privatepagecount / 1gb
$w3wpMemory += $server + ":" + $w3wpresult + "`n"
}
$i = 0
ForEach($server in $serverList)
{
$w3wpresult2 = (get-wmiobject Win32_Process -filter "commandline like '%serviceoptimization%'" -computername $server).privatepagecount / 1gb
Write-Host $w3wpresult2 " , " ($w3wpresult | select-object -index $i)
if($w3wpresult -contains ($w3wpresult2))
{
$toBeRecycled += $server + "`r`n"
}
$i = $i + 1
}
$toBeRecycled += "*******************************************************************************"
$toBeRecycled += "`r`n"
Write-Host $toBeRecycled
out-file -filepath $filepath -Append -inputobject $toBeRecycled -encoding ASCII -width 100
return $toBeRecycled
When the script runs, I see the output of from the execution of the other scripts.
The results from the "Invoke-Expression" command are returning null, why is this?
Write-Host writes directly to the host display. If you want to capture this output then use Write-Output instead or just put a variable on a line by itself because the default output is the "output" stream:
$toBeRecycled
BTW when you execute a PowerShell script from another PowerShell script, the child script will execute synchronously (unless you are using jobs).
I had made an initial question here Which was answered but as i move along in my task I'm running into another problem.
Summary: I have a log file that's being written to via a serial device. I'm wanting to monitor this log file for particular strings (events) and when they happen i want to write those strings to a separate file.
Executing this one off does what I'm looking for:
$p = #("AC/BATT_PWR","COMM-FAULT")
$fileName = "SRAS_$(Get-Date -format yyyy-MM-dd).log"
$fullPath = "C:\temp\SRAS\$fileName"
Get-Content $fullpath -tail 1 -Wait | Select-String -Pattern $p -SimpleMatch | Out-File -Filepath C:\temp\SRAS\sras_pages.log -Append
The problem is the logfile gets a datestamp, putty saves it as SRAS_yyyy-mm-dd.log. So when the clock passes midnight this will no longer be looking at the correct file.
I found this post on SO which is exactly what I want to do, the OP claims it works for him. I modified it slightly for my purposes but it doesn't write events matching the desired strings to sras_pages.log
This is the 'modified' code:
while($true)
{
$now = Get-Date
$fileName = "SRAS_$(Get-Date -format yyyy-MM-dd).log"
$fullPath = "C:\temp\SRAS\$fileName"
$p = #("AC/BATT_PWR","COMM-FAULT")
Write-Host "[$(Get-Date)] Starting job for file $fullPath"
$latest = Start-Job -Arg $fullPath -ScriptBlock {
param($file)
# wait until the file exists, just in case
while(-not (Test-Path $fullpath)){ sleep -sec 10 }
Get-Content $file -Tail 1 -wait | Select-String -Pattern $p |
foreach { Out-File -Filepath "C:\temp\SRAS\sras_pages.log" -Append }
}
# wait until day changes, or whatever would cause new log file to be created
while($now.Date -eq (Get-Date).Date){ sleep -Sec 10 }
# kill the job and start over
Write-Host "[$(Get-Date)] Stopping job for file $fullPath"
$latest | Stop-Job
}
If I execute just the Get-Content segment of that code it does exactly what I'm looking for. I can't figure out what the issue is.
TIA for advice.
Here is a few suggested changes that should make it work:
$p does not exist within the job, add it as a parameter ($pattern in my example)
You are referring to $fullpath within your job (row 13), it should be $file.
Add parameter -SimpleMatch to select-string to search for literal strings instead of regular expressions. (This is not needed but will come in handy if you change search pattern)
Referring to $pattern instead of $p (see 1)
Skip the foreach on row 16.
Like this:
while($true)
{
$now = Get-Date
$fileName = "SRAS_$(Get-Date -format yyyy-MM-dd).log"
$fullPath = "C:\temp\SRAS\$fileName"
$p = #("AC/BATT_PWR","COMM-FAULT")
Write-Host "[$(Get-Date)] Starting job for file $fullPath"
$latest = Start-Job -Arg $fullPath, $p -ScriptBlock {
param($file,$pattern)
# wait until the file exists, just in case
while(-not (Test-Path $file)){ sleep -sec 10 }
Get-Content $file -Tail 1 -wait | Select-String -Pattern $pattern -SimpleMatch |
Out-File -Filepath "C:\temp\SRAS\sras_pages.log" -Append
}
# wait until day changes, or whatever would cause new log file to be created
while($now.Date -eq (Get-Date).Date){ sleep -Sec 10 }
# kill the job and start over
Write-Host "[$(Get-Date)] Stopping job for file $fullPath"
$latest | Stop-Job
}