How to tail log files in Powershell for particular string - powershell

We would like to tail several log files "live" in powershell for particular string. We have tried to use "get-content" command but without luck because everytime as a result we received only results from one file. I assume that it was caused by "-wait" parameter. If there is any other way to tail multiple files ? I heard about runspaces but still I don't know how to implement it.
Our sample script below
dir d:\test\*.txt -include *.txt | `
Get-Content -Wait | `
select-string "windows" | `
ForEach-Object {Write-EventLog -LogName Application -Source "Application error" -EntryType information -EventId 999 -Message $_}
Any help will be appreciated.

Might be slightly late to the party...
As per here, we can use the following in a .ps1 file;
$ProgressPreference='SilentlyContinue'
Workflow My-Tail
{
Param([string[]] $Path)
foreach -parallel ($file in $path)
{
Get-Content -Path $file -Tail 1 -Wait
}
}
This lets us tail multiple files, invoking in powershell like;
My-Tail (dir C:\Users\User\Documents\*.log)
All that's left then is to filter by the required string;
My-Tail (dir C:\Users\User\Documents\*.log) | Select-String -Pattern ".*ERROR.*"
This will tail all files ending in .log in Documents, outputting lines that contain the text 'ERROR'

The only way I can think to do this is to create a bunch of jobs that run the command on each file. You can then receive the job output in a loop and changes in any of the files will be written to the screen:
gci d:\test\*.txt | % { $sb = [scriptblock]::create("get-content -wait $_") ; start-job -ScriptBlock $sb }
while(1) {
$m = $(get-job | receive-job | select-string "searchstring")
if($m -ne $null) {
Write-EventLog -LogName Application -Source "Application error" -EntryType information -EventId 999 -Message $m
}
sleep 1
}

Related

How do I ship logs from SharePoint in almost real time to a fileshare using PowerShell?

I've got a SharePoint farm where I'm trying to ship the log files in "real time" to a server that is available for the monitoring team using PowerShell.
I first had it going pretty well using Get-SPLogEvent, until i noticed that using the cmdlet in it self produced log entries. And I was going about it in 1 second intervalls :O
So back to my original idea with using Get-Content -Wait then.
I put the log shipping in a job which is aborted when a newer log file is created.
This works reasonably well except when the logfile I'm trying to ship is to big to start with.
Most often, that is the case. The first log file I try to ship is empty, and the shipping is started only with the second log file.
Is there a way to have Get-Content -Wait work correctly in a pipe with files as large as 75 - 100 MB?
$SourceLogFolder = <somewhere local>
$LogShipFolder = <somewhere remote>
while ($true) {
$NewLog = Get-ChildItem $SourceLogFolder | Select -Last 1 #Find the latest log file
if ($NewLog.Name -ne $CurrentLog.Name) {#the current log file has been closed or is new
$CurrentLog = $NewLog
Get-Job | Remove-Job -Force #clear any previous log shippings
Start-Job {#Tail CurrentLog
Get-Content $Using:CurrentLog.FullName -Wait |
Out-File "$($Using:LogShipFolder)\$($Using:CurrentLog.Name)" -Encoding utf8
}#end job
}#end if
sleep -seconds 30
}#end while
Ok, for some reason Out-File sometimes creates the new file once before using it when piping large files.
So using Out-File -Force will have it remove the 0 byte large shipped log file that was created.
[CmdletBinding()]
param (
[Parameter(Mandatory)][ValidateNotNullOrEmpty()]
[string]$SourceLogFolder,
[Parameter(Mandatory)][ValidateNotNullOrEmpty()]
[string]$LogShipFolder,
[int]$CheckInterval = 5, #seconds
[int]$CleanupTimeSpan = 3 #hours
)
$SourceLogFolder = $SourceLogFolder.TrimEnd('\')
$LogShipFolder = $LogShipFolder.TrimEnd('\')
$ShippedLogName = 'null'
while ($true) {
$NewLog = Get-ChildItem $SourceLogFolder | Select -Last 1
$ShippedLog = Get-ChildItem $ShippedLogName -ErrorAction SilentlyContinue
if (#the current log was closed or is new
$NewLog.Name -ne $CurrentLog.Name -and
$ShippedLog.Length -eq $ShippedLogJustNow.Length
) {
$CurrentLog = $NewLog
$ShippedLogName = "$LogShipFolder\$($CurrentLog.Name)"
Get-Job | Remove-Job -Force #kill previous log shipping
Start-ThreadJob {#Tail Current log
Get-Content ($Using:CurrentLog).FullName -Wait |
Out-File $Using:ShippedLogName -Encoding utf8 -Force
}#end job
Start-ThreadJob {#Cleanup shipped logs
$YoungestFileFound = 1
$OldLogFiles = Get-ChildItem $Using:LogShipFolder -Recurse |
where LastWriteTime -le (get-date).AddHours(-$Using:CleanupTimeSpan) |
Select -SkipLast $YoungestFileFound
$OldLogFiles | Remove-Item -Recurse -Force -Confirm:$false
}#end job
}#end if
$ShippedLogJustNow = Get-ChildItem $ShippedLogName -ErrorAction SilentlyContinue
sleep -Seconds $CheckInterval
[system.gc]::Collect() #Garbage collect to minimize memory leaks
}#end while

Finding and running a file

I am creating a script where it searches through a bunch of nested folders to find the newest version of a software, which are in .msi form. My code currently can find the file and output it, but is not able to run the file.
I can use Select in the last line for the ForEach to output the correct file but when I change it to Start-Process, I get bombarded by errors.
$path="S:\\Releases\\Program"
$NoOfDirs=Get-ChildItem $path -Directory
ForEach($dir in $NoOfDirs){
Get-ChildItem "$path\$($dir.name)" -File -Recurse |
Where-Object {$_.LastWriteTime -gt ([DateTime]::Now.Adddays(-1))} |
Select-Object #{l='Folder';e={$dir.Name}},Name,LastWriteTime |
Sort-Object -pro LastWriteTime -Descending |
Start-Process -First 1
}
Is there a different command I should be using when running .msi files?
Since your code has to "search through a bunch of nested folders", I'd recommend using the -Recurse switch on Get-ChildItem.
Also use the -Filter parameter to have the search limited to .MSI files.
Something like this:
$path = "S:\Releases\Program"
$refDate = (Get-Date).Adddays(-1)
Get-ChildItem -Path $path -Filter '*.msi' -File -Recurse |
Where-Object {$_.LastWriteTime -gt $refDate} |
ForEach-Object {
# create the arguments for msiexec.exe.
# to find out more switches, open a commandbox and type msiexec /?
$msiArgs = '/i', ('"{0}"' -f $_.FullName), '/qn', '/norestart'
$exitCode = Start-Process 'msiexec.exe' -ArgumentList $msiArgs -NoNewWindow -Wait -PassThru
if ($exitCode -ne 0) {
# something went wrong. see http://www.msierrors.com/tag/msiexec-return-codes/
# to find out what the error was.
Write-Warning "MsiExec.exe returned error code $exitCode"
}
}

Powershell tail multiple files command

I can tail one file via the following command:
Get-Content -Path C:\log1.txt -Tail 10 –Wait
How do I extend this to multiple files, I have tried the following with no luck:
Get-Content -Path C:\log1.txt,C:\log2.txt -Tail 10 –Wait
This will only pick up updates from the first file, not the second.
Based on #mjolinor's comment, I have come up with the following that appears to work,
Workflow My-Tail
{
Param([string[]] $Path)
foreach -parallel ($file in $path)
{
Get-Content -Path $file -Tail 1 -Wait
}
}
My-Tail (dir C:\*.log -Include log1.txt,log2.txt)
However, this has some sort of progress bar that appears...
I can't speak to how efficient this is, but since I'm using PowerShell Core 7.1.3, I can't use Workflows or ForEach -Parallel, but I can use ForEach-Object -Parallel, so I tried it just to see what would happen...
gci -Path C:\ -Filter log*.txt |
% -Parallel {
cat -Wait -Tail 10 -Path $_
} -ThrottleLimit 30
In my case, I had 27 files I needed to monitor, so I chose a number just above that, and this seemed to work.
Just to be sure it was working, I used this, which will output the source file name before each line:
gci -Path C:\ -Filter log*.txt |
% -Parallel {
$file = $_;
cat -Wait -Tail 10 -Path $file |
% { write "$($file.Name): ${_}" }
} -ThrottleLimit 30
I needed tailed output across multiple files and I wanted to try do it in one line,
here's what I eventually came up with:
gci *.txt -recurse | ForEach-Object { Write-Output "$_`n" + $(Get-Content $_ -tail 5) + "`n" }
Its takes a recursive directory listing of all files named *.txt,
writes the file path to console,
then writes the last 5 lines to console.
I didn't need to follow the tails of the files, they weren't being actively written to.

Powershell - Monitoring live log file Q2

I had made an initial question here Which was answered but as i move along in my task I'm running into another problem.
Summary: I have a log file that's being written to via a serial device. I'm wanting to monitor this log file for particular strings (events) and when they happen i want to write those strings to a separate file.
Executing this one off does what I'm looking for:
$p = #("AC/BATT_PWR","COMM-FAULT")
$fileName = "SRAS_$(Get-Date -format yyyy-MM-dd).log"
$fullPath = "C:\temp\SRAS\$fileName"
Get-Content $fullpath -tail 1 -Wait | Select-String -Pattern $p -SimpleMatch | Out-File -Filepath C:\temp\SRAS\sras_pages.log -Append
The problem is the logfile gets a datestamp, putty saves it as SRAS_yyyy-mm-dd.log. So when the clock passes midnight this will no longer be looking at the correct file.
I found this post on SO which is exactly what I want to do, the OP claims it works for him. I modified it slightly for my purposes but it doesn't write events matching the desired strings to sras_pages.log
This is the 'modified' code:
while($true)
{
$now = Get-Date
$fileName = "SRAS_$(Get-Date -format yyyy-MM-dd).log"
$fullPath = "C:\temp\SRAS\$fileName"
$p = #("AC/BATT_PWR","COMM-FAULT")
Write-Host "[$(Get-Date)] Starting job for file $fullPath"
$latest = Start-Job -Arg $fullPath -ScriptBlock {
param($file)
# wait until the file exists, just in case
while(-not (Test-Path $fullpath)){ sleep -sec 10 }
Get-Content $file -Tail 1 -wait | Select-String -Pattern $p |
foreach { Out-File -Filepath "C:\temp\SRAS\sras_pages.log" -Append }
}
# wait until day changes, or whatever would cause new log file to be created
while($now.Date -eq (Get-Date).Date){ sleep -Sec 10 }
# kill the job and start over
Write-Host "[$(Get-Date)] Stopping job for file $fullPath"
$latest | Stop-Job
}
If I execute just the Get-Content segment of that code it does exactly what I'm looking for. I can't figure out what the issue is.
TIA for advice.
Here is a few suggested changes that should make it work:
$p does not exist within the job, add it as a parameter ($pattern in my example)
You are referring to $fullpath within your job (row 13), it should be $file.
Add parameter -SimpleMatch to select-string to search for literal strings instead of regular expressions. (This is not needed but will come in handy if you change search pattern)
Referring to $pattern instead of $p (see 1)
Skip the foreach on row 16.
Like this:
while($true)
{
$now = Get-Date
$fileName = "SRAS_$(Get-Date -format yyyy-MM-dd).log"
$fullPath = "C:\temp\SRAS\$fileName"
$p = #("AC/BATT_PWR","COMM-FAULT")
Write-Host "[$(Get-Date)] Starting job for file $fullPath"
$latest = Start-Job -Arg $fullPath, $p -ScriptBlock {
param($file,$pattern)
# wait until the file exists, just in case
while(-not (Test-Path $file)){ sleep -sec 10 }
Get-Content $file -Tail 1 -wait | Select-String -Pattern $pattern -SimpleMatch |
Out-File -Filepath "C:\temp\SRAS\sras_pages.log" -Append
}
# wait until day changes, or whatever would cause new log file to be created
while($now.Date -eq (Get-Date).Date){ sleep -Sec 10 }
# kill the job and start over
Write-Host "[$(Get-Date)] Stopping job for file $fullPath"
$latest | Stop-Job
}

PowerShell run script simultaneously

I created a PowerShell script to remove all files and folders older than X days. This works perfectly fine and the logging is also ok. Because PowerShell is a bit slow, it can take some time to delete these files and folders when big quantities are to be treated.
My questions: How can I have this script ran on multiple directories ($Target) at the same time?
Ideally, we would like to have this in a scheduled task on Win 2008 R2 server and have an input file (txt, csv) to paste some new target locations in.
Thank you for your help/advise.
The script
#================= VARIABLES ==================================================
$Target = \\share\dir1"
$OlderThanDays = "10"
$Logfile = "$Target\Auto_Clean.log"
#================= BODY =======================================================
# Set start time
$StartTime = (Get-Date).ToShortDateString()+", "+(Get-Date).ToLongTimeString()
Write-Output "`nDeleting folders that are older than $OlderThanDays days:`n" | Tee-Object $LogFile -Append
Get-ChildItem -Directory -Path $Target |
Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-$OlderThanDays) } | ForEach {
$Folder = $_.FullName
Remove-Item $Folder -Recurse -Force -ErrorAction SilentlyContinue
$Timestamp = (Get-Date).ToShortDateString()+" | "+(Get-Date).ToLongTimeString()
# If folder can't be removed
if (Test-Path $Folder)
{ "$Timestamp | FAILLED: $Folder (IN USE)" }
else
{ "$Timestamp | REMOVED: $Folder" }
} | Tee-Object $LogFile -Append # Output folder names to console & logfile at the same time
# Set end time & calculate runtime
$EndTime = (Get-Date).ToShortDateString()+", "+(Get-Date).ToLongTimeString()
$TimeTaken = New-TimeSpan -Start $StartTime -End $EndTime
# Write footer to log
Write-Output ($Footer = #"
Start Time : $StartTime
End Time : $EndTime
Total runtime : $TimeTaken
$("-"*79)
"#)
# Create logfile
Out-File -FilePath $LogFile -Append -InputObject $Footer
# Clean up variables at end of script
$Target=$StartTime=$EndTime=$OlderThanDays = $null
One way to achieve this would be to write an "outer" script that passes the directory-to-be-cleaned, into the "inner" script, as a parameter.
For your "outer" script, have something like this:
$DirectoryList = Get-Content -Path $PSScriptRoot\DirList;
foreach ($Directory in $DirectoryList) {
Start-Process -FilePath powershell.exe -ArgumentList ('"{0}\InnerScript.ps1" -Path "{1}"' -f $PSScriptRoot, $Directory);
}
Note: Using Start-Process kicks off a new process that is, by default, asynchronous. If you use the -Wait parameter, then the process will run synchronously. Since you want things to run more quickly and asynchronously, omitting the -Wait parameter should achieve the desired results.
Invoke-Command
Alternatively, you could use Invoke-Command to kick off a PowerShell script, using the parameters: -File, -ArgumentList, -ThrottleLimit, and -AsJob. The Invoke-Command command relies on PowerShell Remoting, so that must enabled, at least on the local machine.
Add a parameter block to the top of your "inner" script (the one you posted above), like so:
param (
[Parameter(Mandatory = $true)]
[string] $Path
)
That way, your "outer" script can pass in the directory path, using the -Path parameter for the "inner" script.