I have a Windows command line utility which outputs text to standard output, as well as logging to a file. The two outputs complement each other, and so I want to be able combine both streams. The standard output will remain untouched. However, I aim to take chunks of the log file, process them, and also send it to standard output.
My first attempt was to run:
[HashTable] $queue = #{};
$roboDealerJob = Start-Job -ScriptBlock {
Param(
[string] $outputFile,
[HashTable] $queue
)
& "utility.exe" $outputFile |
ForEach-Object
{
$queue.Add($_, $_);
}
} -Name "MyUtility" -ArgumentList $outputFile,$queue;
While (!(Test-Path $logFile))
{
Start-Sleep -Milliseconds 100;
}
Get-Content -Path $logFile -Tail 1 -Wait |
ForEach-Object {
Receive-Job $roboDealerJob
ForEach ($item in $queue.Values)
{
Write-Host $item;
$queue.Remove($item);
}
Write-Host $_;
}
However, the job doesn't seem to run.
Any suggestions on how to do this?
Related
I am using Powershell 7.
We have the following PowerShell script that will parse some very large file.
I no longer want to use 'Get-Content' as this is to slow.
The script below works, but it takes a very long time to process even a 10 MB file.
I have about 200 files 10MB file with over 10000 lines.
Sample Log:
#Fields:1
#Fields:2
#Fields:3
#Fields:4
#Fields: date-time,connector-id,session-id,sequence-number,local-endpoint,remote-endpoint,event,data,context
2023-01-31T13:53:50.404Z,EXCH1\Relay-EXCH1,08DAD23366676FF1,41,10.10.10.2:25,195.85.212.22:15650,<,DATA,
2023-01-31T13:53:50.404Z,EXCH1\Relay-EXCH1,08DAD23366676FF1,41,10.10.10.2:25,195.85.212.25:15650,<,DATA,
Script:
$Output = #()
$LogFilePath = "C:\LOGS\*.log"
$LogFiles = Get-Item $LogFilePath
$Count = #($logfiles).count
ForEach ($Log in $LogFiles)
{
$Int = $Int + 1
$Percent = $Int/$Count * 100
Write-Progress -Activity "Collecting Log details" -Status "Processing log File $Int of $Count - $LogFile" -PercentComplete $Percent
Write-Host "Processing Log File $Log" -ForegroundColor Magenta
Write-Host
$FileContent = Get-Content $Log | Select-Object -Skip 5
ForEach ($Line IN $FileContent)
{
$Socket = $Line | Foreach {$_.split(",")[5] }
$IP = $Socket.Split(":")[0]
$Output += $IP
}
}
$Output = $Output | Select-Object -Unique
$Output = $Output | Sort-Object
Write-Host "List of noted remove IPs:"
$Output
Write-Host
$Output | Out-File $PWD\Output.txt
As #iRon Suggests the assignment operator (+=) is a lot of overhead. As well as reading entire file to a variable then processing it. Perhaps process it strictly as a pipeline. I achieved same results, using your sample data, with the code written this way below.
$LogFilePath = "C:\LOGS\*.log"
$LogFiles = Get-ChildItem $LogFilePath
$Count = #($logfiles).count
$Output = ForEach($Log in $Logfiles) {
# Code for Write-Progress here
Get-Content -Path $Log.FullName | Select-Object -Skip 5 | ForEach-Object {
$Socket = $_.split(",")[5]
$IP = $Socket.Split(":")[0]
$IP
}
}
$Output = $Output | Select-Object -Unique
$Output = $Output | Sort-Object
Write-Host "List of noted remove IPs:"
$Output
Apart from the notable points in the comments, I believe this question is more suitable to Code Review. Nonetheless, here's my take on this using the StreamReader class:
$LogFilePath = "C:\LOGS\*.log"
$LogFiles = Get-Item -Path $LogFilePath
$OutPut = [System.Collections.ArrayList]::new()
foreach ($log in $LogFiles)
{
$skip = 0
$stop = $false
$stream = [System.IO.StreamReader]::new($log.FullName)
while ($line = $stream.ReadLine())
{
if (-not$stop)
{
if ($skip++ -eq 5)
{
$stop = $true
}
continue
}
elseif ($OutPut.Contains(($IP = ($line -split ',|:')[6])))
{
continue
}
$null = $OutPut.Add($IP)
}
$stream.Close()
$stream.Dispose()
}
# Display OutPut and save to file
Write-Host -Object "List of noted remove IPs:"
$OutPut | Sort-Object | Tee-Object -FilePath "$PWD\Output.txt"
This way you can output unique IP's since it's being handled by an if statement checking against what's in $OutPut; essentially replacing Select-Object -Unique. You should see a speed increase as you're no longer adding to a fixed array (+=), and piping to other cmdlets.
You can combine File.ReadLines with Enumerable.Skip to read your files and skip their first 5 lines. This method is much faster than Get-Content. Then for sorting and getting unique strings at the same time you can use a SortedSet<T>.
You should avoid using Write-Progress as this will slow your script down in Windows PowerShell (this has been fixed in newer versions of PowerShell Core).
Do note that because you're looking to sort the result, all strings must be contained in memory before outputting to a file. This would be much more efficient if sorting was not needed, there you would use a HashSet<T> instead for getting unique values.
Get-Item C:\LOGS\*.log | & {
begin { $set = [Collections.Generic.SortedSet[string]]::new() }
process {
foreach($line in [Linq.Enumerable]::Skip([IO.File]::ReadLines($_.FullName), 5)) {
$null = $set.Add($line.Split(',')[5].Split(':')[0])
}
}
end {
$set
}
} | Set-Content $PWD\Output.txt
i have the following function in my script
function Write-Host($object)
{
if($global:LogFile -eq $null)
{
$global:LogFile = $logFile
}
$object | tee $global:LogFile -Append
}
referencing this post: https://stackoverflow.com/a/25847258/8397835
I am trying specifically this part here:
$job = Start-Job -ScriptBlock { Start-Sleep -Seconds 10 }
while (($job.State -eq "Running") -and ($job.State -ne "NotStarted"))
{
Write-Host ([char]9632) -NoNewLine
Start-Sleep -Seconds 1
}
apparently, with tee, nonewline appears to be ignored...and without tee, i am getting the characters to display on one line as i am seeking
with tee:
without tee
I think i know whats happening. since write-host is being converted to tee, any switches are ignored, be it color or in this case, nonewline. How can i make nonewline work with tee?
After our chat I understand what you're trying to do. You want to write yourself a custom progress bar that both writes to a log file as well as to the console without line breaks in either. For that you can write a function that will accomplish it, but I do recommend picking a new name that doesn't conflict with an existing cmdlet. I'll use Write-MyProgress.
Function Write-MyProgress{
[cmdletbinding()]
Param(
[parameter(valuefrompipeline=$true)]$message,
[switch]$NoNewLine
)
if($global:LogFile -eq $null)
{
$global:LogFile = $logFile
}
Add-Content -Value $message -Path $LogFile -NoNewline:$NoNewLine
Write-Host $Message -NoNewLine:$NoNewLine
}
You could then call it explicitly:
Write-MyProgress ([char]9632) -NoNewLine
or pipe things to it:
[char]9632 | Write-MyProgress -NoNewLine
Or, if you don't want to use a function, you could just do it all with native cmdlets like in this example:
1..10 | ForEach-Object -Process {
[char]9632 | Add-Content $LogFile -NoNewLine -PassThru | Write-Host -NoNewLine
start-sleep -Sec 1
} -End {Add-Content -Value '' -Path $LogFile}
(Note that I add '' to the log file at the end, so the log file gets a new line after the progress bar is done)
I had made an initial question here Which was answered but as i move along in my task I'm running into another problem.
Summary: I have a log file that's being written to via a serial device. I'm wanting to monitor this log file for particular strings (events) and when they happen i want to write those strings to a separate file.
Executing this one off does what I'm looking for:
$p = #("AC/BATT_PWR","COMM-FAULT")
$fileName = "SRAS_$(Get-Date -format yyyy-MM-dd).log"
$fullPath = "C:\temp\SRAS\$fileName"
Get-Content $fullpath -tail 1 -Wait | Select-String -Pattern $p -SimpleMatch | Out-File -Filepath C:\temp\SRAS\sras_pages.log -Append
The problem is the logfile gets a datestamp, putty saves it as SRAS_yyyy-mm-dd.log. So when the clock passes midnight this will no longer be looking at the correct file.
I found this post on SO which is exactly what I want to do, the OP claims it works for him. I modified it slightly for my purposes but it doesn't write events matching the desired strings to sras_pages.log
This is the 'modified' code:
while($true)
{
$now = Get-Date
$fileName = "SRAS_$(Get-Date -format yyyy-MM-dd).log"
$fullPath = "C:\temp\SRAS\$fileName"
$p = #("AC/BATT_PWR","COMM-FAULT")
Write-Host "[$(Get-Date)] Starting job for file $fullPath"
$latest = Start-Job -Arg $fullPath -ScriptBlock {
param($file)
# wait until the file exists, just in case
while(-not (Test-Path $fullpath)){ sleep -sec 10 }
Get-Content $file -Tail 1 -wait | Select-String -Pattern $p |
foreach { Out-File -Filepath "C:\temp\SRAS\sras_pages.log" -Append }
}
# wait until day changes, or whatever would cause new log file to be created
while($now.Date -eq (Get-Date).Date){ sleep -Sec 10 }
# kill the job and start over
Write-Host "[$(Get-Date)] Stopping job for file $fullPath"
$latest | Stop-Job
}
If I execute just the Get-Content segment of that code it does exactly what I'm looking for. I can't figure out what the issue is.
TIA for advice.
Here is a few suggested changes that should make it work:
$p does not exist within the job, add it as a parameter ($pattern in my example)
You are referring to $fullpath within your job (row 13), it should be $file.
Add parameter -SimpleMatch to select-string to search for literal strings instead of regular expressions. (This is not needed but will come in handy if you change search pattern)
Referring to $pattern instead of $p (see 1)
Skip the foreach on row 16.
Like this:
while($true)
{
$now = Get-Date
$fileName = "SRAS_$(Get-Date -format yyyy-MM-dd).log"
$fullPath = "C:\temp\SRAS\$fileName"
$p = #("AC/BATT_PWR","COMM-FAULT")
Write-Host "[$(Get-Date)] Starting job for file $fullPath"
$latest = Start-Job -Arg $fullPath, $p -ScriptBlock {
param($file,$pattern)
# wait until the file exists, just in case
while(-not (Test-Path $file)){ sleep -sec 10 }
Get-Content $file -Tail 1 -wait | Select-String -Pattern $pattern -SimpleMatch |
Out-File -Filepath "C:\temp\SRAS\sras_pages.log" -Append
}
# wait until day changes, or whatever would cause new log file to be created
while($now.Date -eq (Get-Date).Date){ sleep -Sec 10 }
# kill the job and start over
Write-Host "[$(Get-Date)] Stopping job for file $fullPath"
$latest | Stop-Job
}
I've got two powershell functions in a file. The first one (Get-ImapAttachments) connects using the NetCmdlets IMAP cmdlets, downloads all *.csv attachments from any messages in the INBOX, and then moves the messages to another folder to keep the INBOX clean. I got this function from the NetCmdlets site, and tweaked it slightly for my needs. On it's own, it works the way I need it to.
I've written a second function (Import-D2LData) to process the CSV file. It's a badly formatted CSV file, complete with a report title on the first line, and the column headings are repeated every 23 lines or so. Since Import-Csv isn't quite as flexible as I need, I use a combination of get-content and select-object -skip 1 to start off the import, and basically build a long string to pass to ConvertFrom-Csv. If I run that function manually, it currently works the way I need it to, also.
The problem arises when I run it in the .ps1 file I've got them in. The first function fires off just fine, and the second one seems to. The ConvertFrom-Csv fails and I end up with a bunch of empty lines of output. The real strangeness is, I can use the . command to source the .ps1 file, and run the last 7 lines of the script (starting with the call to Get-ImapAttachments), and I get the expected results.
I'm sure I'm doing something pretty braindead, but after staring at the code for a couple of hours, I'm just not seeing it at this point...
# Needed for the IMAP Stuff
Add-PSSnapin -Name NetCmdlets -ErrorAction SilentlyContinue
function Get-ImapAttachments {
param(
[string] $server,
[string] $user,
[string] $password,
[string] $downloadDir="c:\temp\attachments",
[string] $folder = "INBOX"
)
$destFolder = "$folder.Processed"
$imapConnection = Connect-Imap -server $server -user $user -password $password
$m_messages = Get-Imap -connection $imapConnection -folder $folder
foreach($msg in $m_messages)
{
if($msg.ContentType.StartsWith("multipart")) #A MIME message with multiple parts
{
for($i=0;$i -lt $msg.PartCount;$i++)
{
if($msg.PartFileName[$i] -like "*.csv" ) #There is a CSV attachment
{
$localFile = $downloadDir + "\" + $msg.PartFileName[$i]
if( Test-Path -Path $localFile ) {
Remove-Item $localFile -Force
}
Get-Imap -connection $imapConnection -folder $folder -view $msg.Id -localfile $localFile -part $msg.PartId[$i]
Move-Imap -Connection $imapConnection -folder $folder -Message $msg.Id -Destination $destFolder
}
}
}
}
Disconnect-IMAP -Connection $imapConnection
}
function Import-D2LData {
param(
[string] $CsvFile
)
Write-Host -ForegroundColor Cyan $CsvFile
$file = get-content $CsvFile | Select-Object -Skip 1
$head = $file[0].Split(',')
$data = "$head`n"
foreach ( $line in $file ) {
if( $line -ne $file[0] ) {
$data += "$line`n"
}
}
$csv = $data | ConvertFrom-Csv -Header $head
$csv
}
Get-ImapAttachments -server imap.contoso.com -user sa_mailbox -password 'password'
$csvFiles = Get-ChildItem -Path "c:\temp\attachments" -Filter "*.csv"
foreach( $file in $csvFiles ) {
Import-D2LData -CsvFile $file.FullName
}
Well, it wasn't pretty, but I ended up breaking the functions into two different scripts. When that wasn't enough, I ran the first script in it's own PowerShell instance, which seemed to isolate the runspaces enough to allow both functions to operate properly:
$BasePath = 'Z:\scripts\Import-Data'
powershell.exe -NoProfile -command "& '$BasePath\fetch-attachments.ps1'"
& "$BasePath\process-attachments.ps1"
Not exactly an elegant solution, but it got me past the issues I was having...
I created a PowerShell script to remove all files and folders older than X days. This works perfectly fine and the logging is also ok. Because PowerShell is a bit slow, it can take some time to delete these files and folders when big quantities are to be treated.
My questions: How can I have this script ran on multiple directories ($Target) at the same time?
Ideally, we would like to have this in a scheduled task on Win 2008 R2 server and have an input file (txt, csv) to paste some new target locations in.
Thank you for your help/advise.
The script
#================= VARIABLES ==================================================
$Target = \\share\dir1"
$OlderThanDays = "10"
$Logfile = "$Target\Auto_Clean.log"
#================= BODY =======================================================
# Set start time
$StartTime = (Get-Date).ToShortDateString()+", "+(Get-Date).ToLongTimeString()
Write-Output "`nDeleting folders that are older than $OlderThanDays days:`n" | Tee-Object $LogFile -Append
Get-ChildItem -Directory -Path $Target |
Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-$OlderThanDays) } | ForEach {
$Folder = $_.FullName
Remove-Item $Folder -Recurse -Force -ErrorAction SilentlyContinue
$Timestamp = (Get-Date).ToShortDateString()+" | "+(Get-Date).ToLongTimeString()
# If folder can't be removed
if (Test-Path $Folder)
{ "$Timestamp | FAILLED: $Folder (IN USE)" }
else
{ "$Timestamp | REMOVED: $Folder" }
} | Tee-Object $LogFile -Append # Output folder names to console & logfile at the same time
# Set end time & calculate runtime
$EndTime = (Get-Date).ToShortDateString()+", "+(Get-Date).ToLongTimeString()
$TimeTaken = New-TimeSpan -Start $StartTime -End $EndTime
# Write footer to log
Write-Output ($Footer = #"
Start Time : $StartTime
End Time : $EndTime
Total runtime : $TimeTaken
$("-"*79)
"#)
# Create logfile
Out-File -FilePath $LogFile -Append -InputObject $Footer
# Clean up variables at end of script
$Target=$StartTime=$EndTime=$OlderThanDays = $null
One way to achieve this would be to write an "outer" script that passes the directory-to-be-cleaned, into the "inner" script, as a parameter.
For your "outer" script, have something like this:
$DirectoryList = Get-Content -Path $PSScriptRoot\DirList;
foreach ($Directory in $DirectoryList) {
Start-Process -FilePath powershell.exe -ArgumentList ('"{0}\InnerScript.ps1" -Path "{1}"' -f $PSScriptRoot, $Directory);
}
Note: Using Start-Process kicks off a new process that is, by default, asynchronous. If you use the -Wait parameter, then the process will run synchronously. Since you want things to run more quickly and asynchronously, omitting the -Wait parameter should achieve the desired results.
Invoke-Command
Alternatively, you could use Invoke-Command to kick off a PowerShell script, using the parameters: -File, -ArgumentList, -ThrottleLimit, and -AsJob. The Invoke-Command command relies on PowerShell Remoting, so that must enabled, at least on the local machine.
Add a parameter block to the top of your "inner" script (the one you posted above), like so:
param (
[Parameter(Mandatory = $true)]
[string] $Path
)
That way, your "outer" script can pass in the directory path, using the -Path parameter for the "inner" script.