Powershell extracting logs based upon interval? - powershell

I have script logic as follows:-
check if the folder is empty
If yes Use the same command with exception time period of last 30 days. i.e StartDate etc
If folder not empty get / extract the logs for 1 day using same command
just tweaking timesettings.
the script will be scheduled to run every 5 minutes. For every run extract using the same command for 1 day saves the result in temp.csv and compare it most updated copy of final-admin.csv.
If changes found; write difference / over-write in
final-admin.csv
code
Add-PSSnapin Microsoft.Exchange.Management.PowerShell.E2010
$outcsv='c:\Mail-audit-results\Final-mail-admin.csv'
if(Get-ChildItem C:\Mail-audit-results){
$enddate=[datetime]::Today
$Startdate=$enddate.AddDays(-1)
$splat=#{
Identity='mail.test-nw.com'
LogonTypes=#('Admin','Delegate')
ShowDetails=$true
StartDate=$Startdate
EndDate=$enddate
}
Search-MailboxAuditLog #splat | Export-Csv c:\Mail-audit-results\temp_results.csv -NoTypeInformation -Append
$Old = Import-CSV $outcsv
$New = Import-CSV c:\Mail-audit-results\temp_results.csv
$New | ?{$Old -notmatch $_} | Export-CSV $outcsv -notype
}
#process items when files found
}else{
# process when no files found.
$enddate=[datetime]::Today
$Startdate=$enddate.AddDays(-30)
$splat=#{
Identity='mail.test-nw.com'
LogonTypes=#('Admin','Delegate')
ShowDetails=$true
StartDate=$Startdate
EndDate=$enddate
}
Search-MailboxAuditLog #splat | Export-Csv $outcsv -NoTypeInformation -Append
}
Get-ChildItem 'c:\Mail-audit-results\Final-mail-admin.csv' | ForEach {
Else
{
return 0;
}
}
Problem-points
How can I change the format of date in mm/dd/yyy format in section
$enddate=[datetime]::Today
$Startdate=$enddate.AddDays(-1)
How to overwrite with difference data?
$New | ?{$Old -notmatch $_} | Export-CSV $outcsv -notype
I don't want to write to third file the difference I want original file $outcsv to get written; would the above command overwrite/ or just append?
Thanks.

You can change your dates using -Uformat switch
$enddate= Get-Date ([datetime]::Today) -UFormat "%m%d%Y"
$Startdate= Get-Date ($enddate.AddDays(-1)) -UFormat "%m%d%Y"
The line
$New | ?{$Old -notmatch $_} | Export-CSV $outcsv -notype
will overwrite data, Export-CSV -Append will append.

Related

Insert Content into specific place in Powershell

I am trying to compare the string of two CSV files. If the string from the 2nd CSV file occurs in the 1st CSV file, the corresponding line in the 1st CSV file should be marked with a label (e.g.: "TestLabel") after the semicolon. The strings contain a lot of special characters. By and large, the comparison already works, I can also already add the label.
Since Powershell is still new to me and this is my first script, the following question still arises. How can I set my text "TestLabel" to a certain place in an uncomplicated way? Here, for example, in the next empty field between the semicolons?
CSV1 contains:
Testdefinition;Stichwörter;Stichwörter;Stichwörter;Stichwörter;Stichwörter
It is just a normal text (with round brackets).Test: success;ExistingLabel;;;;
This is a second text;;;
Another text;ExistingLabel;;;;
One more text for the testing - success;ExistingLabel;;;;
CSV2 contains:
Testdefinition;Stichwörter;Stichwörter;Stichwörter;Stichwörter;Stichwörter
It is just a normal text (with round brackets).Test: success
One more text for the testing - success
My script so far:
$header='Testdefinition', 'Stichwörter1', 'Stichwörter2', 'Stichwörter3', 'Stichwörter4', 'Stichwörter5'
$exportheader="Testdefinition;Stichwörter;Stichwörter;Stichwörter;Stichwörter;Stichwörter"
$path1='D:\data\.....test.csv'
$path2='D:\data\.....test_failed.csv'
$wfile='temp1.csv'
$wfile2='temp2.csv'
Get-Content $path1 | Select-Object -Skip 1 | Set-Content $wfile -Encoding UTF8
Get-Content $path2 | Select-Object -Skip 1 | Set-Content $wfile2 -Encoding UTF8
$file1=Import-CSV -Path $wfile -Delimiter ";" -Header $header
$file2=Import-CSV -Path $wfile2 -Delimiter ";" -Header $header
$exportfile='test.csv'
#$exportfile=$file1
$file1 | Get-Member
$file2 | Get-Member
$file1 | Format-Table
$file2 | Format-Table
Write-Output ""
Write-Output "Searching for failed results"
Set-Content $exportfile -Value $exportheader
$file1.Testdefinition | ForEach-Object {
Write-Output "The Testdefinition is: $_ "
$testSearch = $_
$testlinecontent = $file2.Testdefinition | Select-String $testSearch
$testlinenumber = $testlinecontent.LineNumber
if("$_" -eq "$testlinecontent")
{
Write-Output "Testline found: $testlinecontent in Line $testlinenumber"
Write-Output "$_ = $testlinecontent"
$testlineexport = "$_;$testlinenumber;TestLabel"
Write-Output $testlineexport
$testlineexport | Add-Content -Path $exportfile
}
else
{
Write-Output "Testline not found"
$testlineexport = "$_;$testlinenumber;NULL"
Write-Output $testlineexport
$testlineexport | Add-Content -Path $exportfile
}
Write-Output ""
}
$exportCsv = Import-Csv $exportfile -Delimiter ";" -Header $header
$exportCsv | Format-Table
Remove-Item -Path $wfile
Remove-Item -Path $wfile2
I hope you can give me a hint. Thanks in advance!
Assuming the files aren't too big, you can use the following approach based on Compare-Object, which is conceptually clear and relatively simple:
# Read the CSV files into their header row and the array of data rows, as strings.
$header, $rows1 = Get-Content $path1
$null, $rows2 = Get-Content $path2
# Initialize the export file by writing its header
Set-Content -Encoding utf8 $exportfile -Value $exportheader
# Compare the data rows by their first ";"-separated field.
# If the fields match, append ";TestLabel" to the LHS data row before
# passing it through, otherwise pass it as-is, and append to the
# export file.
Compare-Object -PassThru $rows1 $rows2 -IncludeEqual -Property { $_.Split(';')[0] } |
ForEach-Object { if ($_.SideIndicator -eq '==') { $_ + ';TestLabel' } else { $_ } } |
Add-Content $exportfile
Note:
For brevity I've omitted the code to also add a line number.
As you are already aware, PowerShell doesn't support CSV files whose headers contain duplicate column names, given that the column names become property names on import, and must therefore be unique.

Ignore round brackets of strings when comparing CSV files [duplicate]

I am trying to compare the string of two CSV files. If the string from the 2nd CSV file occurs in the 1st CSV file, the corresponding line in the 1st CSV file should be marked with a label (e.g.: "TestLabel") after the semicolon. The strings contain a lot of special characters. By and large, the comparison already works, I can also already add the label.
Since Powershell is still new to me and this is my first script, the following question still arises. How can I set my text "TestLabel" to a certain place in an uncomplicated way? Here, for example, in the next empty field between the semicolons?
CSV1 contains:
Testdefinition;Stichwörter;Stichwörter;Stichwörter;Stichwörter;Stichwörter
It is just a normal text (with round brackets).Test: success;ExistingLabel;;;;
This is a second text;;;
Another text;ExistingLabel;;;;
One more text for the testing - success;ExistingLabel;;;;
CSV2 contains:
Testdefinition;Stichwörter;Stichwörter;Stichwörter;Stichwörter;Stichwörter
It is just a normal text (with round brackets).Test: success
One more text for the testing - success
My script so far:
$header='Testdefinition', 'Stichwörter1', 'Stichwörter2', 'Stichwörter3', 'Stichwörter4', 'Stichwörter5'
$exportheader="Testdefinition;Stichwörter;Stichwörter;Stichwörter;Stichwörter;Stichwörter"
$path1='D:\data\.....test.csv'
$path2='D:\data\.....test_failed.csv'
$wfile='temp1.csv'
$wfile2='temp2.csv'
Get-Content $path1 | Select-Object -Skip 1 | Set-Content $wfile -Encoding UTF8
Get-Content $path2 | Select-Object -Skip 1 | Set-Content $wfile2 -Encoding UTF8
$file1=Import-CSV -Path $wfile -Delimiter ";" -Header $header
$file2=Import-CSV -Path $wfile2 -Delimiter ";" -Header $header
$exportfile='test.csv'
#$exportfile=$file1
$file1 | Get-Member
$file2 | Get-Member
$file1 | Format-Table
$file2 | Format-Table
Write-Output ""
Write-Output "Searching for failed results"
Set-Content $exportfile -Value $exportheader
$file1.Testdefinition | ForEach-Object {
Write-Output "The Testdefinition is: $_ "
$testSearch = $_
$testlinecontent = $file2.Testdefinition | Select-String $testSearch
$testlinenumber = $testlinecontent.LineNumber
if("$_" -eq "$testlinecontent")
{
Write-Output "Testline found: $testlinecontent in Line $testlinenumber"
Write-Output "$_ = $testlinecontent"
$testlineexport = "$_;$testlinenumber;TestLabel"
Write-Output $testlineexport
$testlineexport | Add-Content -Path $exportfile
}
else
{
Write-Output "Testline not found"
$testlineexport = "$_;$testlinenumber;NULL"
Write-Output $testlineexport
$testlineexport | Add-Content -Path $exportfile
}
Write-Output ""
}
$exportCsv = Import-Csv $exportfile -Delimiter ";" -Header $header
$exportCsv | Format-Table
Remove-Item -Path $wfile
Remove-Item -Path $wfile2
I hope you can give me a hint. Thanks in advance!
Assuming the files aren't too big, you can use the following approach based on Compare-Object, which is conceptually clear and relatively simple:
# Read the CSV files into their header row and the array of data rows, as strings.
$header, $rows1 = Get-Content $path1
$null, $rows2 = Get-Content $path2
# Initialize the export file by writing its header
Set-Content -Encoding utf8 $exportfile -Value $exportheader
# Compare the data rows by their first ";"-separated field.
# If the fields match, append ";TestLabel" to the LHS data row before
# passing it through, otherwise pass it as-is, and append to the
# export file.
Compare-Object -PassThru $rows1 $rows2 -IncludeEqual -Property { $_.Split(';')[0] } |
ForEach-Object { if ($_.SideIndicator -eq '==') { $_ + ';TestLabel' } else { $_ } } |
Add-Content $exportfile
Note:
For brevity I've omitted the code to also add a line number.
As you are already aware, PowerShell doesn't support CSV files whose headers contain duplicate column names, given that the column names become property names on import, and must therefore be unique.

powershell foreach shows duplicate result

I use powershell to automate extracting of selected data from a CSV file.
My $target_servers also contains two the same server name but it has different data in each rows.
Here is my code:
$target_servers = Get-Content -Path D:\Users\Tools\windows\target_prd_servers.txt
foreach($server in $target_servers) {
Import-Csv $path\Serverlist_Template.csv | Where-Object {$_.Hostname -Like $server} | Export-Csv -Path $path/windows_prd.csv -Append -NoTypeInformation
}
After executing the above code it extracts CSV data based on a TXT file, but my problem is some of the results are duplicated.
I am expecting around 28 results but it gave me around 49.
As commented, -Append is the culprit here and you should check if the newly added records are not already present in the output file:
# read the Hostname column of the target csv file as array to avoid duplicates
$existingHostsNames = #((Import-Csv -Path "$path/windows_prd.csv").Hostname)
$target_servers = Get-Content -Path D:\Users\Tools\windows\target_prd_servers.txt
foreach($server in $target_servers) {
Import-Csv "$path\Serverlist_Template.csv" |
Where-Object {($_.Hostname -eq $server) -and ($existingHostsNames -notcontains $_.HostName)} |
Export-Csv -Path "$path/windows_prd.csv" -Append -NoTypeInformation
}
You can convert your data to array of objects and then use select -Unique, like this:
$target_servers = Get-Content -Path D:\Users\Tools\windows\target_prd_servers.txt
$data = #()
foreach($server in $target_servers) {
$data += Import-Csv $path\Serverlist_Template.csv| Where-Object {$_.Hostname -Like $server}
}
$data | select -Unique | Export-Csv -Path $path/windows_prd.csv -Append -NoTypeInformation
It will work only if duplicated rows have same value in every column. If not, you can pass column names to select which are important for you. For ex.:
$data | select Hostname -Unique | Export-Csv -Path $path/windows_prd.csv -Append -NoTypeInformation
It will give you list of unique hostnames.

Parse csv along comparing txt file lined and log lines not found

As a continuation of a script I'm running, working on the following.
I have a CSV file that has formatted information, example as follows:
File named Import.csv:
Name,email,x,y,z
\I\RS\T\Name1\c\x,email#jksjks,d,f
\I\RS\T\Name2\d\f,email#jsshjs,d,f
...
This file is large.
I also have another file called Note.txt.
Name1
Name2
Name3
...
With help from #mathias-r-jessen
$Dir = PathToFile
$import = Import-Csv $Dir\import.csv
$NoteFile = "$Dir\Note.txt"
$Note = GC $NoteFile
$Import |Where-Object {$Note -contains $_.Name.Split('\')[4]} |Export-Csv "$Dir\Result.csv" -NoTypeInformation -Append
This code quickly and effortlessly parses the big csv and extracts every line that contains any of the lines in the $note file.
My next question is how do i log any lines in the $note file that were not found in the csv file.
I tried the following:
$result = $Import |Where-Object {$Note -contains $_.Name.Split('\')[4]} |Export-Csv "$Dir\Result.csv" -NoTypeInformation -Append
$Note | Where-Object {$result.Name.Split('\')[4] -notcontains $Note} | out-file $dir\not-found.log -append
This seems to return every line in $note.
#mathias-r-jessen any help you can provide would be appreciated.
You could use a Switch to do that.
Switch($Import){
{$Note -contains $_.Name.Split('\')[4]} {$_ | Export-Csv "$Dir\Result.csv" -NoTypeInformation -Append; continue}
default {$_ | Export-csv "$Dir\Not-Found.csv" -NoType -Append}
}
The continue in the first option makes it so that if the first case is a match it performs the relevant action, and then continues to the next record. If the first case doesn't match it moves on to the default action, which outputs it to a different file.
I solved it by using the following:
$result = $Import |Where-Object {$Note -contains $_.Name.Split('\')[4]}
$result | Export-Csv "$Dir\Result.csv" -NoTypeInformation -Append
$matches = $note | where-object { $result.Name -match $_}
compare-object $note $matches |where-object {$_.SideIndicator -like "<=" | select -ExpandProperty InputObject | Out-file "$Dir\Not_found.txt" -Append

Splitting CSV file by two columns

Starting with a 500,000 line CSV, I need to split the files by day and hour (the second and third columns). I've tried the modify the group to include the hour and while I see the hour get added to my filename, I get no results in the exported file.
The foreach doing the work:
foreach ($group in $data | Group Day,hour) {
$data | Where-Object { $_.Day -and $_.Hour -eq $group.Name }
ConvertTo-Csv -NoTypeInformation |
foreach {$_.Replace('"','')} |
Out-File "$Path\Testfile_$($group.name -replace $regexA, '').csv"
Sample Data:
Bob,1/27/2012,8:00,Basic,Operations
Charlie,2/3/2012,9:00,Advanced,Production
Bill,3/7/2012,10:00,Advanced,Production
You could import the CSV, determine the output filename on the fly, and append each record to the matchning file:
Import-Csv 'C:\path\to\input.csv' | ForEach-Object {
$filename = ('output_{0}_{1}.csv' -f $_.Day, $_.Hour) -replace '[/:]'
$_ | Export-Csv "C:\path\to\$filename" -Append -NoType
}
Note that Export-Csv -Append requires PowerShell v3 or newer.