I am using Powershell 7.
We have the following PowerShell script that will parse some very large file.
I no longer want to use 'Get-Content' as this is to slow.
The script below works, but it takes a very long time to process even a 10 MB file.
I have about 200 files 10MB file with over 10000 lines.
Sample Log:
#Fields:1
#Fields:2
#Fields:3
#Fields:4
#Fields: date-time,connector-id,session-id,sequence-number,local-endpoint,remote-endpoint,event,data,context
2023-01-31T13:53:50.404Z,EXCH1\Relay-EXCH1,08DAD23366676FF1,41,10.10.10.2:25,195.85.212.22:15650,<,DATA,
2023-01-31T13:53:50.404Z,EXCH1\Relay-EXCH1,08DAD23366676FF1,41,10.10.10.2:25,195.85.212.25:15650,<,DATA,
Script:
$Output = #()
$LogFilePath = "C:\LOGS\*.log"
$LogFiles = Get-Item $LogFilePath
$Count = #($logfiles).count
ForEach ($Log in $LogFiles)
{
$Int = $Int + 1
$Percent = $Int/$Count * 100
Write-Progress -Activity "Collecting Log details" -Status "Processing log File $Int of $Count - $LogFile" -PercentComplete $Percent
Write-Host "Processing Log File $Log" -ForegroundColor Magenta
Write-Host
$FileContent = Get-Content $Log | Select-Object -Skip 5
ForEach ($Line IN $FileContent)
{
$Socket = $Line | Foreach {$_.split(",")[5] }
$IP = $Socket.Split(":")[0]
$Output += $IP
}
}
$Output = $Output | Select-Object -Unique
$Output = $Output | Sort-Object
Write-Host "List of noted remove IPs:"
$Output
Write-Host
$Output | Out-File $PWD\Output.txt
As #iRon Suggests the assignment operator (+=) is a lot of overhead. As well as reading entire file to a variable then processing it. Perhaps process it strictly as a pipeline. I achieved same results, using your sample data, with the code written this way below.
$LogFilePath = "C:\LOGS\*.log"
$LogFiles = Get-ChildItem $LogFilePath
$Count = #($logfiles).count
$Output = ForEach($Log in $Logfiles) {
# Code for Write-Progress here
Get-Content -Path $Log.FullName | Select-Object -Skip 5 | ForEach-Object {
$Socket = $_.split(",")[5]
$IP = $Socket.Split(":")[0]
$IP
}
}
$Output = $Output | Select-Object -Unique
$Output = $Output | Sort-Object
Write-Host "List of noted remove IPs:"
$Output
Apart from the notable points in the comments, I believe this question is more suitable to Code Review. Nonetheless, here's my take on this using the StreamReader class:
$LogFilePath = "C:\LOGS\*.log"
$LogFiles = Get-Item -Path $LogFilePath
$OutPut = [System.Collections.ArrayList]::new()
foreach ($log in $LogFiles)
{
$skip = 0
$stop = $false
$stream = [System.IO.StreamReader]::new($log.FullName)
while ($line = $stream.ReadLine())
{
if (-not$stop)
{
if ($skip++ -eq 5)
{
$stop = $true
}
continue
}
elseif ($OutPut.Contains(($IP = ($line -split ',|:')[6])))
{
continue
}
$null = $OutPut.Add($IP)
}
$stream.Close()
$stream.Dispose()
}
# Display OutPut and save to file
Write-Host -Object "List of noted remove IPs:"
$OutPut | Sort-Object | Tee-Object -FilePath "$PWD\Output.txt"
This way you can output unique IP's since it's being handled by an if statement checking against what's in $OutPut; essentially replacing Select-Object -Unique. You should see a speed increase as you're no longer adding to a fixed array (+=), and piping to other cmdlets.
You can combine File.ReadLines with Enumerable.Skip to read your files and skip their first 5 lines. This method is much faster than Get-Content. Then for sorting and getting unique strings at the same time you can use a SortedSet<T>.
You should avoid using Write-Progress as this will slow your script down in Windows PowerShell (this has been fixed in newer versions of PowerShell Core).
Do note that because you're looking to sort the result, all strings must be contained in memory before outputting to a file. This would be much more efficient if sorting was not needed, there you would use a HashSet<T> instead for getting unique values.
Get-Item C:\LOGS\*.log | & {
begin { $set = [Collections.Generic.SortedSet[string]]::new() }
process {
foreach($line in [Linq.Enumerable]::Skip([IO.File]::ReadLines($_.FullName), 5)) {
$null = $set.Add($line.Split(',')[5].Split(':')[0])
}
}
end {
$set
}
} | Set-Content $PWD\Output.txt
I have a script that is supposed to look at a logfile that starts filling with several hundred lines. While the log is filling, I need to look for two different conditions at the same time. If I find "Completed Successfully" then it will break the while loop and continue with the rest of the script. If it finds "Error" in the script, it will restart the server. Unfortunately this isn't working for me. It just sits until the timeout of the stopwatch. I'm not quite sure what is wrong here. Is there a better way to do this?
$Comp="ServerA.domain"
$Logfile="\\$Comp\Logs\logfile1.txt"
$pattern1 = "Completed Successfully"
$Pattern2 = "Error"
$timeout = new-timespan -Minutes 5
$stopwatch = [diagnostics.stopwatch]::StartNew()
while ($stopwatch.elapsed -lt $timeout){
try{
$logContent2 = Get-Content -Path $Logfile -Tail 1000 | Select-String -Pattern $Pattern2 | Measure-Object | Select-Object -ExpandProperty Count
$logContent1 = Get-Content -Path $Logfile -Tail 1000 | select-string -pattern $pattern1 | Measure-Object | Select-Object -ExpandProperty Count
If ($logContent1 -gt 0) {
Write-Host -Object "Compiled Successfully on $Comp"
#break;
}
ElseIf ($logContent2 -gt 0) {
Write-Host -Object "Error Found on $Comp! Restart required. Rebooting..."
Restart-Computer -ComputerName $Comp -Wait -For PowerShell -Force
}
}
Catch{
$Error[0] > \\$Comp\Logs\ErrorLog.txt
}
}
If ($stopwatch.elapsed -ge $timeout){
Write-Error -Message "$pattern1 did not appear" -ErrorAction Stop
exit;
}
Yes! I accidentally placed a break in the wrong place. This is resolved. The above code works but you need to break the loop (I had a break in my code in the wrong place) and I don't include the break in the code in this example.
Hi i have powershell where i am using a foreach-object and would like to skip the first iteration always. And i am using continue statement as well. But the current behaviour of continue is like break. Please suggest if something i am doing wrong here.
Below is the sample code.
$xmlfile = 'D:\testdirecotry\sample.xml'
[xml]$xmlcontent = (Get-Content $xmlfile)
$folderprefix = 'plm_z'
$regex = '<!--__AMAZONSITE id="(.+?)" instance="(.+?)"__-->'
$i=0
(Get-Content $xmlfile) | select-string -Pattern $regex | ForEach-Object {
write-host "Test Iterartion"
if($i -eq 0)
{
write-host "entering if loop"
write-host $i
$i++
write-host $i
continue
}
else
{
write-host "entering else loop"
write-host $_
$pscustomobject=#(
# write-host $_
$id = $_.Matches.Groups[1].Value
$instance = $_.Matches.Groups[2].Value
write-host "Do Something"
)
}
}
An easier way to skip the first object would potentially be to use Select-Object in the pipeline:
Get-Content $xmlfile |
Select-string -Pattern $regex |
Select-Object -Skip 1 |
ForEach-Object {
...
i have the following function in my script
function Write-Host($object)
{
if($global:LogFile -eq $null)
{
$global:LogFile = $logFile
}
$object | tee $global:LogFile -Append
}
referencing this post: https://stackoverflow.com/a/25847258/8397835
I am trying specifically this part here:
$job = Start-Job -ScriptBlock { Start-Sleep -Seconds 10 }
while (($job.State -eq "Running") -and ($job.State -ne "NotStarted"))
{
Write-Host ([char]9632) -NoNewLine
Start-Sleep -Seconds 1
}
apparently, with tee, nonewline appears to be ignored...and without tee, i am getting the characters to display on one line as i am seeking
with tee:
without tee
I think i know whats happening. since write-host is being converted to tee, any switches are ignored, be it color or in this case, nonewline. How can i make nonewline work with tee?
After our chat I understand what you're trying to do. You want to write yourself a custom progress bar that both writes to a log file as well as to the console without line breaks in either. For that you can write a function that will accomplish it, but I do recommend picking a new name that doesn't conflict with an existing cmdlet. I'll use Write-MyProgress.
Function Write-MyProgress{
[cmdletbinding()]
Param(
[parameter(valuefrompipeline=$true)]$message,
[switch]$NoNewLine
)
if($global:LogFile -eq $null)
{
$global:LogFile = $logFile
}
Add-Content -Value $message -Path $LogFile -NoNewline:$NoNewLine
Write-Host $Message -NoNewLine:$NoNewLine
}
You could then call it explicitly:
Write-MyProgress ([char]9632) -NoNewLine
or pipe things to it:
[char]9632 | Write-MyProgress -NoNewLine
Or, if you don't want to use a function, you could just do it all with native cmdlets like in this example:
1..10 | ForEach-Object -Process {
[char]9632 | Add-Content $LogFile -NoNewLine -PassThru | Write-Host -NoNewLine
start-sleep -Sec 1
} -End {Add-Content -Value '' -Path $LogFile}
(Note that I add '' to the log file at the end, so the log file gets a new line after the progress bar is done)
I am looking for some help in reading latest entry in a file for two strings and then save that name do an un installation of that driver .I wrote this for searching but seems to return true for every instance.
$filename = "\\SigCap.log"
$md = "Device is missing"
$DeviceArray = "Sigcap"
$ErrStrng = [Regex]::Matches($filename,$md) | Measure-Object -Maximum Index
$Devstrng = [Regex]::Matches($filename,$DeviceArray) | Measure-Object -Maximum Index
If($Devstrng.Maximum -eq $ErrStrng.Maximum)
{
write-output "yes"
Write-host $DeviceArray
}
else
{
write-output "no"
Write-host $DeviceArray
}
Ok, you're using [regex]::Matches() incorrectly, but I'm not going to go there right now. Instead use the -Match operator in a Where statement since it uses RegEx by default. When used with the Get-Content command using the -Tail parameter this could be simple.
$LastLine = Get-Content $File -Tail 1
If($LastLine -match $md -and $LastLine -match $DeviceArray){
"yes"
}Else{
"no"
}
Write-host $DeviceArray
I think that's what you're trying to accomplish.
Ok, after reading your comments, I think this will do what you want:
$filename = "\\SigCap.log"
$md = "Device is missing"
$DeviceArray = "Sigcap"
$ErrStrng = select-string -Path $filename -Pattern $md |select -last 1 -expand LineNumber
$Devstrng = select-string -Path $filename -Pattern $devicearray |select -last 1 -expand LineNumber
If($Devstrng -eq $ErrStrng)
{
write-output "yes"
Write-host $DeviceArray
}
else
{
write-output "no"
Write-host $DeviceArray
}