I am trying to Monitor a Log file using Powershell, but I am not able to figure out what regex should I use to have my required Monitoring output.
Get-Content $file -wait | where {$_ -match "some regex"} |
foreach { send_email($_) }
Here is a sample of my Log File.
19:43:06.5230 Info {"message":"YourCode_Prod execution started","level":"Information","timeStamp":"2019-01-15T19:43:06.5132404+00:00","fingerprint":"588aeb19-76e5-415a-88ff-a69797eb414f","windowsIdentity":"AD\\gaurav.m","machineName":"EDEV-3","processName":"YourCode_Prod","processVersion":"1.0.6800.16654","fileName":"YourCode_Work","jobId":"22a537ae-35e6-4abd-a57d-9dd0c273e81a","robotName":"Gaurav"}
19:50:48.8014 Info {"message":"YourCode_Prod execution ended","level":"Information","timeStamp":"2019-01-15T19:50:48.8005228+00:00","fingerprint":"b12b7d6f-cf3a-4e24-b1e6-1cf4413c12e2","windowsIdentity":"AD\\gaurav.m","machineName":"EDEV-3","processName":"YourCode_Prod","processVersion":"1.0.6800.16654","fileName":"YourCode_Work","jobId":"22a537ae-35e6-4abd-a57d-9dd0c273e81a","robotName":"Gaurav","totalExecutionTimeInSeconds":462,"totalExecutionTime":"00:07:42"}
I tried to generate regex, but I could not find the right logic for this.
$searchpattern = [regex]"(?:(?:started))|(?:(?:ended))"
Get-Content -Path C:\Execution.log | foreach {
if ($PSItem -match $searchpattern) {
# does the entry have a valid date/time
$time, $type, $data = $PSItem -split " ", 3
$time
$type
$data
}
}
I need to monitor this file which will have many lines, but I want to take action only for "execution started" and "execution ended".
I could separate the type of message time and the data. But I need to further drill down on the data.
This script will run every 5 min, so I will compare time -5 min and start reading log from there, as soon as I find ended or started I will take necessary action.
Your log data is in JSON format, so you could simply process the extracted data as such.
Get-Content -Path C:\Execution.log | ForEach-Object {
$time, $type, $json = $_ -split " ", 3
$data = ConvertFrom-Json $json
if ($data.message -match 'execution (started|ended)') {
# do stuff
}
}
Related
I want to do this
read the file
go through each line
if the line matches the pattern, do some changes with that line
save the content to another file
For now I use this script:
$file = [System.IO.File]::ReadLines("C:\path\to\some\file1.txt")
$output = "C:\path\to\some\file2.txt"
ForEach ($line in $file) {
if($line -match 'some_regex_expression') {
$line = $line.replace("some","great")
}
Out-File -append -filepath $output -inputobject $line
}
As you can see, here I write line by line. Is it possible to write the whole file at once ?
Good example is provided here :
(Get-Content c:\temp\test.txt) -replace '\[MYID\]', 'MyValue' | Set-Content c:\temp\test.txt
But my problem is that I have additional IF statement...
So, what could I do to improve my script ?
You could do it like that:
Get-Content -Path "C:\path\to\some\file1.txt" | foreach {
if($_ -match 'some_regex_expression') {
$_.replace("some","great")
}
else {
$_
}
} | Out-File -filepath "C:\path\to\some\file2.txt"
Get-Content reads a file line by line (array of strings) by default so you can just pipe it into a foreach loop, process each line within the loop and pipe the whole output into your file2.txt.
In this case Arrays or Array List(lists are better for large arrays) would be the most elegant solution. Simply add strings in array until ForEach loop ends. After that just flush array to a file.
This is Array List example
$file = [System.IO.File]::ReadLines("C:\path\to\some\file1.txt")
$output = "C:\path\to\some\file2.txt"
$outputData = New-Object System.Collections.ArrayList
ForEach ($line in $file) {
if($line -match 'some_regex_expression') {
$line = $line.replace("some","great")
}
$outputData.Add($line)
}
$outputData |Out-File $output
I think the if statement can be avoided in a lot of cases by using regular expression groups (e.g. (.*) and placeholders (e.g. $1, $2 etc.).
As in your example:
(Get-Content .\File1.txt) -Replace 'some(_regex_expression)', 'great$1' | Set-Content .\File2.txt
And for the good example" where [MYID\] might be somewhere inline:
(Get-Content c:\temp\test.txt) -Replace '^(.*)\[MYID\](.*)$', '$1MyValue$2' | Set-Content c:\temp\test.txt
(see also How to replace first and last part of each line with powershell)
Guys i'm having some issues converting my Perl script to powershell, I need some help. In the host file of our machines, we have all of the URL's to our test environments blocked. In my PERL script, based on which environment is selected, it will comment out the line of the environment selected to allow access and block others so the testers can't mistakenly do things in the wrong environment.
I need help converting to powershell
Below is what I have in PERL:
sub editHosts {
print "Editing hosts file...\n";
my $file = 'C:\\Windows\\System32\\Drivers\\etc\\hosts';
my $data = readFile($file);
my #lines = split /\n/, $data;
my $row = '1';
open (FILE, ">$file") or die "Cannot open $file\n";
foreach my $line (#lines) {
if ($line =~ m/$web/) {
print FILE '#'."$line\n"; }
else {
if ($row > '21') {
$line =~ s/^\#*127\.0\.0\.1/127\.0\.0\.1/;
$line =~ s/[#;].*$//s; }
print FILE "$line\n"; }
$row++;
}
close(FILE);
}
Here is what i've tried in Powershell:
foreach ($line in get-content "C:\windows\system32\drivers\etc\hosts") {
if ($line -contains $web) {
$line + "#"
}
I've tried variation including set-content with what used to be in the host file, etc.
Any help would be appreciated!
Thanks,
Grant
-contains is a "set" operator, not a substring operator. Try .Contains() or -like.
This will comment out lines matching the variable $word, while removing # from non-matches (except the header):
function Edit-Hosts ([string]$Web, $File = "C:\windows\system32\drivers\etc\hosts") {
#If file exists and $web is not empty/whitespace
if((Test-Path -Path $file -PathType Leaf) -and $web.Trim()) {
$row = 1
(Get-Content -Path $file) | ForEach-Object {
if($_ -like "*$web*") {
#Matched PROD, comment out line
"#$($_)"
} else {
#No match. If past header = remove comment
if($row -gt 21) { $_ -replace '^#' } else { $_ }
}
$row++
} | Set-Content -Path $file
} else {
Write-Error -Category InvalidArgument -Message "'$file' doesn't exist or Web-parameter is empty"
}
}
Usage:
Edit-Hosts -Web "PROD"
This is a similar answer to Frode F.'s answer, but I'm not yet able to comment to add my 2c worth, so have to provide an alternative answer instead.
It looks like one of the gotchas moving from perl to PowerShell, in this example, is that when we get the content of the file using Get-Content it is an "offline" copy, i.e. any edits are not made directly to the file itself. One approach is to compile the new content to the whole file and then write that back to disk.
I suppose that the print FILE "some text\n"; construct in perl might be similar to "some text" | Out-File $filename -Encoding ascii -Append in PowerShell, albeit you would use the latter either (1) to write line-by-line to a new/empty file or (2) accept that you are appending to existing content.
Two other things about editing the hosts file:
Be sure to make sure that your hosts file is ASCII encoded; I have caused a major outage for a key enterprise application (50k+ users) in learning that...
You may need to remember to run your PowerShell / PowerShell ISE by right-clicking and choosing Run as Administrator else you might not be able to modify the file.
Anyway, here's a version of the previous answer using Out-File:
$FileName = "C:\windows\system32\drivers\etc\hosts"
$web = "PROD"
# Get "offline" copy of file contents
$FileContent = Get-Content $FileName
# The following creates an empty file and returns a file
# object (type [System.IO.FileInfo])
$EmptyFile = New-Item -Path $FileName -ItemType File -Force
foreach($Line in $FileContent) {
if($Line -match "$web") {
"# $Line" | Out-File $EmptyFile -Append -Encoding ascii
} else {
"$Line" | Out-File $EmptyFile -Append -Encoding ascii
}
}
Edit
The ($Line -match "$web") takes whatever is in the $web variable and treats it as a regular expression. In my example I was assuming that you were just wanting to match a simple text string, but you might well be trying to match an IP address, etc. You have a couple of options:
Use ($Line -like "*$web*") instead.
Convert what is in $web to be an escaped regex, i.e. one that will match literally. Do this with ($Line -match [Regex]::Escape($web)).
You also wanted to strip off comments from any line past row 21 of the hosts file, should that line not match $web. In perl you have used the s substitution operator; the PowerShell equivalent is -replace.
So... here is an updated version of that foreach loop:
$LineCount = 1
foreach($Line in $FileContent) {
if($Line -match [Regex]::Escape($web) {
# ADD comment to any matched line
$Line = "#" + $Line
} elseif($LineCount -gt 21) {
# Uncomment the other lines
$Line = $Line -replace '^[# ]+',''
}
# Remove 'stacked up' comment characters, if any
$Line = $Line -replace '[#]+','#'
$Line | Out-File $EmptyFile -Append -Encoding ascii
$LineCount++
}
More Information
Are there good references for moving from Perl to Powershell?
How to use operator '-replace' in PowerShell to replace strings of texts with special characters and replace successfully
about_Comparison_Operators
http://www.comp.leeds.ac.uk/Perl/sandtr.html
If you wanted to verify what was in there and then add entries, you could use the below which is designed to be ran interactively and returns any existing entries you specify in the varibles:
Note: the `t is powershell's in script method for 'Tab' command.
$hostscontent
# Script to Verify and Add Host File Entries
$hostfile = gc 'C:\Windows\System32\drivers\etc\hosts'
$hostscontent1 = $hostfile | select-string "autodiscover.XXX.co.uk"
$hostscontent2 = $hostfile | select-string "webmail.XXX.co.uk"
$1 = "XX.XX.XXX.XX`tautodiscover.XXX.co.uk"
$2 = "webmail.XXX.co.uk"
# Replace this machines path with a path to your list of machines e.g. $machines = gc \\machine\machines.txt
$machines = gc 'c:\mytestmachine.txt'
ForEach ($machine in $machines) {
If ($hostscontent1 -ne $null) {
Start-Sleep -Seconds 1
Write-Host "$machine Already has Entry $1" -ForegroundColor Green
} Else {
Write-Host "Adding Entry $1 for $machine" -ForegroundColor Green
Start-Sleep -Seconds 1
Add-Content -Path C:\Windows\System32\drivers\etc\hosts -Value "XX.XX.XXX.XX`tautodiscover.XXX.co.uk" -Force
}
If ($hostscontent2 -ne $null) {
Start-Sleep -Seconds 1
Write-Host "$machine Already has Entry $2" -ForegroundColor Green
} Else {
Write-Host "Adding Entry $2 for $machine" -ForegroundColor Green
Start-Sleep -Seconds 1
Add-Content -Path C:\Windows\System32\drivers\etc\hosts -Value "XX.XX.XXX.XX`twebmail.XXX.co.uk" -Force
}
}
I recently finished my script with the help of someone on this site (Matt) Thanks again!
I now need to somehow get the logfile into a tabled format and I'm not sure how to implement that with the current setup of the script, any ideas?
Write-Host Report generated at (Get-date)
write-host("Lower Environments Status Check");
# Preprocessing Items
$msg = ""
$array = get-content C:\LowerEnvChecklist\appurls.txt
$log = "C:\LowerEnvChecklist\lowerenvironmentslog.txt"
$errorTexts = "error has occurred","Oops","Unable to display widget data","unexpected error occurred","temporarily unavailable","there was a problem"
$regex = ($errorTexts | ForEach-Object{[regex]::Escape($_)}) -join "|"
write-host("Checking appurls.txt...One moment please.");
("`n---------------------------------------------------------------------------") | out-file $log -Append
Get-Date | Out-File $log -Append
("`n***Checking Links***") | out-file $log -Append
("`n") | out-file $log -Append
# Loop through each element of the array.
ForEach($target in $array){
# Erase results for the next pass in case of error.
$result, $response, $stream, $page = $null
# Navigate to site urls
$result = [System.Net.WebRequest]::Create($target)
$response = $result.GetResponse()
$stream = [System.IO.StreamReader]$response.GetResponseStream()
$page = $stream.ReadToEnd()
# To ensure login/authentication pages that give a 403 response pages still show as online
If($response.StatusCode -eq 403){
$msg = " $target -----> is ONLINE!"}
# Determine if the status code 200 pages are truly up based on the information above.
If($response.StatusCode -eq 200){
# While the page might have rendered need to determine there are no errors present.
If($page -notmatch $regex){
$msg = " $target -----> is ONLINE!"
} else {
$msg = " $target -----> may be DOWN, please check!"
}
} else {
$msg = " $target -----> may be DOWN, please check!"
}
# Log Results.
$msg | Out-File $log -Append -width 120
write-host $msg
# Close the response.
$response.Close()
}
# Write completion to logfile.
("`n") | out-file $log -Append
("`n***Lower Environments Checklist Completed***") | out-file $log -Append
# Write completion to host.
write-host("Lower Environments Checklist Completed");
# Open logfile once script is complete.
Invoke-Item C:\LowerEnvChecklist\lowerenvironmentslog.txt
If you just want to view it in-script you could do Out-GridView on your log file. This will open a new window with a view of the data in the log file that looks like a table. Depending on your formatting you may have to add extra items like headers that are human readable.
To wet your whistle with structured output I opted to show you a CSV based solution. Either way all avenues require objects. What we do here is create a custom object that we populate as the script progresses. Each pass sends the details down the pipe. Using the pipeline we can use Export-CSV to collect all of the data in a nice file. Even filtering is possible now.
write-host("Lower Environments Status Check");
# Preprocessing Items
$array = Get-Content C:\LowerEnvChecklist\appurls.txt
$log = "C:\LowerEnvChecklist\lowerenvironmentslog.csv"
$errorTexts = "error has occurred","Oops","Unable to display widget data","unexpected error occurred","temporarily unavailable","there was a problem"
$regex = ($errorTexts | ForEach-Object{[regex]::Escape($_)}) -join "|"
# Loop through each element of the array. Use the pipeline to make output easier
$array | ForEach-Object{
# Keep the variable $target so it is not lost in scopes. Build the object to be completed as we go.
$target = [pscustomobject][ordered]#{
URL = $_
Status = ""
Detail = "N/A"
Timestamp = Get-Date
}
# Erase results for the next pass in case of error.
$result, $response, $stream, $page = $null
# Navigate to site urls. If we cant access the site set a flag to mark the site as down.
$result = [System.Net.WebRequest]::Create($target.URL)
$response = try{$result.GetResponse()}catch{$null}
switch([int]$response.StatusCode){
403{
$target.Status = "OK"
$target.Detail = "403"
}
200{
# Get page content to confirm up status
$stream = [System.IO.StreamReader]$response.GetResponseStream()
$page = $stream.ReadToEnd()
# While the page might have rendered need to determine there are no errors present.
If($page -notmatch $regex){
$target.Status = "OK"
} else {
$target.Status = "DOWN"
$target.Detail = "Pattern"
}
}
default{
$target.Status = "DOWN"
}
}
# Send the object down the pipeline
$target
# Close the response. The object might not exist so check before we call the methods.
if($response){$response.Close()}
if($stream){$stream.Close()}
} | Export-CSV -Path $log -NoTypeInformation
# Write completion to host.
write-host("Lower Environments Checklist Completed");
# Open logfile once script is complete.
Invoke-Item $log
I took the liberty off adding another column to your request called Detail it could add context. Not sure what you wanted from the date but if you have plenty of URLS and processing time then I suppose it could be of use. Also to reduce the if logic I added a switch statement. This would be more useful if you react to other status codes down the road. Still, good thing to know.
Sample Output
URL Status Detail Timestamp
--- ------ ------ ---------
https://7fweb DOWN N/A 1/11/2016 12:18:16 PM
http://www.google.ca OK N/A 1/11/2016 12:18:16 PM
http://www.microsoft.com DOWN Pattern 1/11/2016 12:18:16 PM
I added "windows" to $errorTexts to trigger a pattern match for microsoft.com
I have wrote the following script to read the CSV file to perform the custom format of output.
Script is below:
$Content = Import-Csv Alert.csv
foreach ($Data in $Content) {
$First = $Data.DisplayName
$Second = $Data.ComputerName
$Third = $Data.Description
$Four = $Data.Name
$Five = $Data.ModifiedBy
$Six = $Data.State
$Seven = $Data.Sev
$Eight = $Data.Id
$Nine = $Data.Time
Write-Host "START;"
Write-Host "my_object="`'$First`'`;
Write-Host "my_host="`'$Second`'`;
Write-Host "my_long_msg="`'$Third`'`;
Write-Host "my_tool_id="`'$Four`'`;
Write-Host "my_owner="`'$Five`'`;
Write-Host "my_parameter="`'$Four`'`;
Write-Host "my_parameter_value="`'$Six`'`;
Write-Host "my_tool_sev="`'$Seven`'`;
Write-Host "my_tool_key="`'$Eight`'`;
Write-Host "msg="`'$Four`'`;
Write-Host "END"
}
The above script executing without any error.
Tried with Out-File and redirection operator in PowerShell to dump the output into a file, but I'm not finding any solution.
Write-Host writes to the console. That output cannot be redirected unless you run the code in another process. Either remove Write-Host entirely or replace it with Write-Output, so that the messages are written to the Success output stream.
Using a foreach loop also requires additional measures, because that loop type doesn't support pipelining. Either run it in a subexpression:
(foreach ($Data in $Content) { ... }) | Out-File ...
or assign its output to a variable:
$output = foreach ($Data in $Content) { ... }
$output | Out-File ...
Another option would be replacing the foreach loop with a ForEach-Object loop, which supports pipelining:
$Content | ForEach-Object {
$First = $_.DisplayName
$Second = $_.ComputerName
...
} | Out-File ...
Don't use Out-File inside the loop, because repeatedly opening the file will perform poorly.
First of all, this is my first question here. I often come here to browse existing topics, but now I'm hung on my own problem. And I didn't found a helpful resource right now. My biggest concern would be, that it won't work in Powershell... At the moment I try to get a small Powershell tool to save me a lot of time. For those who don't know cw-sysinfo, it is a tool that collects information of any host system (e.g. Hardware-ID, Product Key and stuff like that) and generates *.txt files.
My point is, if you have 20, 30 or 80 server in a project, it is a huge amount of time to browse all files and just look for those lines you need and put them together in a *.csv file.
What I have working is more like the basic of the tool, it browses all *.txt in a specific path and checks for my keywords. And here is the problem that I just can use the words prior to those I really need, seen as follow:
Operating System: Windows XP
Product Type: Professional
Service Pack: Service Pack 3
...
I don't know how I can tell Powershell to search for "Product Type:"-line and pick the following "Professional" instead. Later on with keys or serial numbers it will be the same problem, that is why I just can't browse for "Standard" or "Professional".
I placed my keywords($controls) in an extra file that I can attach the project folders and don't need to edit in Powershell each time. Code looks like this:
Function getStringMatch
{
# Loop through the project directory
Foreach ($file In $files)
{
# Check all keywords
ForEach ($control In $controls)
{
$result = Get-Content $file.FullName | Select-String $control -quiet -casesensitive
If ($result -eq $True)
{
$match = $file.FullName
# Write the filename according to the entry
"Found : $control in: $match" | Out-File $output -Append
}
}
}
}
getStringMatch
I think this is the kind of thing you need, I've changed Select-String to not use the -quiet option, this will return a matches object, one of the properties of this is the line I then split the line on the ':' and trim any spaces. These results are then placed into a new PSObject which in turn is added to an array. The array is then put back on the pipeline at the end.
I also moved the call to get-content to avoid reading each file more than once.
# Create an array for results
$results = #()
# Loop through the project directory
Foreach ($file In $files)
{
# load the content once
$content = Get-Content $file.FullName
# Check all keywords
ForEach ($control In $controls)
{
# find the line containing the control string
$result = $content | Select-String $control -casesensitive
If ($result)
{
# tidy up the results and add to the array
$line = $result.Line -split ":"
$results += New-Object PSObject -Property #{
FileName = $file.FullName
Control = $line[0].Trim()
Value = $line[1].Trim()
}
}
}
}
# return the results
$results
Adding the results to a csv is just a case of piping the results to Export-Csv
$results | Export-Csv -Path "results.csv" -NoTypeInformation
If I understand your question correctly, you want some way to parse each line from your report files and extract values for some "keys". Here are a few lines to give you an idea of how you could proceede. The example is for one file, but can be generalized very easily.
$config = Get-Content ".\config.txt"
# The stuff you are searching for
$keys = #(
"Operating System",
"Product Type",
"Service Pack"
)
foreach ($line in $config)
{
$keys | %{
$regex = "\s*?$($_)\:\s*(?<value>.*?)\s*$"
if ($line -match $regex)
{
$value = $matches.value
Write-Host "Key: $_`t`tValue: $value"
}
}
}