I have two CSV files I need to compare. Both of which may (or may not) contain an additional delimited field (delimited via a "|" character), like so:
(new.csv)
Title,Key,Value
Jason,Son,Hair=Red|Eyes=Blue
James,Son,Hair=Brown|Eyes=Green
Ron,Father,Hair=Black
Susan,Mother,Hair=Black|Eyes=Brown|Dress=Green
(old.csv)
Title,Key,Value
Jason,Son,Hair=Purple|Eyes=Blue
James,Son,Hair=Brown|Eyes=Green
Ron,Father,Hair=Purple
Susan,Mother,Hair=Black|Eyes=Brown|Dress=Blue
My problem comes in when I attempt to compare the two files...
$fileNew = "new.csv"
$fileOld = "old.csv"
$fileDiffOutputFile = "diff.txt"
$csvNewLog = (Import-CSV ($fileNew))
$csvOldLog = (Import-CSV ($fileOld))
$varDifferences = Compare-Object $csvOldLog $csvNewLog -property Title,Value
$varDifferences | Group-Object -Property Title | % {New-Object -TypeName PSObject -Property #{ NewValue=($_.group[0].Value); Title=$_.name; OldValue=($_.group[1].Value) } } | Out-File $fileDiffOutputFile -Append
Resulting in this output:
(diff.txt)
OldValue Title NewValue
-------- ----- --------
Hair=Purple|Eyes=Blue Jason Hair=Red|Eyes=Blue
Hair=Purple Ron Hair=Black
Hair=Black|Eyes=Brown|D... Susan Hair=Black|Eyes=Brown|...
Some of the values are inevitably going to extend out past the max length of the column, as it does with Susan above.
So, my question could have a couple of solutions that I can think of:
Is there an easier way to isolate the values so that I only pull out the changed values, and not the entire string of delimited values?
If not, is it possible to get the format to show the entire string (including the unchanged values part of the delimited string) instead?
If you include a format-table -wrap in your last line, like so?
$fileNew = "new.csv"
$fileOld = "old.csv"
$fileDiffOutputFile = "diff.txt"
$csvNewLog = (Import-CSV ($fileNew))
$csvOldLog = (Import-CSV ($fileOld))
$varDifferences = Compare-Object $csvOldLog $csvNewLog -property Title,Value
$varDifferences | Group-Object -Property Title | % {New-Object -TypeName PSObject -Property #{ NewValue=($_.group[0].Value); Title=$_.name; OldValue=($_.group[1].Value) } } | Format-Table -wrap | Out-File $fileDiffOutputFile -Append
Related
I am having below data in my csv
"Path_Name","Lun_Number","status"
"vmhba0:C2:T0:L1","1","active"
"vmhba0:C1:T0:L1","1","active"
"vmhba1:C0:T7:L230","230","active"
"vmhba1:C0:T7:L231","231","active"
"vmhba1:C0:T7:L232","230","active"
"vmhba1:C0:T7:L235","231","active"
"vmhba1:C0:T7:L236","230","active"
I need to group the data based on Lun_Number and create a column to get the count of those Lun_Number
expected output
"Path_Name","Lun_Number","status","Count"
"vmhba0:C2:T0:L1","1","active", 2
"vmhba0:C1:T0:L1","1","active",
"vmhba1:C0:T7:L230","230","active",3
"vmhba1:C0:T7:L231","230","active",
"vmhba1:C0:T7:L232","230","active",
"vmhba1:C0:T7:L235","231","active",2
"vmhba1:C0:T7:L236","231","active",
Please let me know how can I do that. I tried group-object, sort-object but it doesn't seems to be working
Below is the code which is generating the above csv
$status_csv = Import-Csv -Path E:\pathstate.csv
$path_csv = Import-Csv -Path E:\PathInfo.csv
foreach($row in $path_csv)
{
$path_1 = $row.Path_Name
$path_2 = $status_csv | where{$_.Name -match "^$path_1$" }
[PsCustomObject]#{
Path_Name = $path_1
Lun_Number = $row.Lun_Number
status = $path_2.PathState
} | Export-Csv -Path E:\FinalReport.csv -NoTypeInformation -Append | Group-Object Lun_Number
}
I can see the approach you're trying to take, and I think something like the following could be useful:
$path_csv = Import-Csv 'sample.csv'
$Unique_Counts = $path_csv.Lun_Number | Group-Object | Select-Object Name,Count
This will help give you the output that you can use as part of your mapping for later, where you can make it dynamic to match with the row you're checking. Which you can use to pull out through a loop, such as $Unique_Counts.
Meaning that it you do something like $Unique_Counts[0].Count, will be able to grab the Lun_Number associated to it (listed as Name in the array).
Name Count
---- -----
1 2
230 3
231 2
If you're okay with having the count for each row, you can then use something like what you have:
foreach($row in $path_csv)
{
$path_1 = $row.Path_Name
[PsCustomObject]#{
Path_Name = $path_1
Lun_Number = $row.Lun_Number
status = $row.status
count = $Unique_Counts | Where-Object {$_.name -eq $row.Lun_Number} | Select-Object -ExpandProperty Count
} | Export-Csv $finalReport -NoTypeInformation -Append
}
This then provides me with the following outcome:
"Path_Name","Lun_Number","status","count"
"vmhba0:C2:T0:L1","1","active","2"
"vmhba0:C1:T0:L1","1","active","2"
"vmhba1:C0:T7:L230","230","active","3"
"vmhba1:C0:T7:L231","231","active","2"
"vmhba1:C0:T7:L232","230","active","3"
"vmhba1:C0:T7:L235","231","active","2"
"vmhba1:C0:T7:L236","230","active","3"
Hope this helps, it may be useful to understand more with the use case, but at least you can grab the unique count for all Lun_numbers. Just has to put it in all rows too.
This question already has an answer here:
Not all properties displayed
(1 answer)
Closed 1 year ago.
This is a follow-up question from PowerShell | EVTX | Compare Message with Array (Like)
I changed the tactic slightly, now I am collecting all the services installed,
$7045 = Get-WinEvent -FilterHashtable #{ Path="1system.evtx"; Id = 7045 } | select
#{N=’Timestamp’; E={$_.TimeCreated.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')}},
Id,
#{N=’Machine Name’; E={$_.MachineName}},
#{N=’Service Name’; E={$_.Properties[0].Value}},#{N=’Image Path’;E=$_.Properties[1].Value}},
#{N=’RunAsUser’; E={$_.Properties[4].Value}},#{N=’Installed By’; E={$_.UserId}}
Now I match each object for any suspicious traits and if found, I add a column 'Suspicious' with the value 'Yes'. This is because I want to leave the decision upto the analyst and pretty sure the bad guys might use something we've not seen before.
foreach ($Evt in $7045)
{
if ($Evt.'Image Path' -match $sus)
{
$Evt | Add-Member -MemberType NoteProperty -Name 'Suspicious' -Value 'Yes'
}
}
Now, I'm unable to get PowerShell to display all columns unless I specifically Select them
$7045 | Format-Table
Same goes for CSV Export. The first two don't include the Suspicious Column but the third one does but that's because I'm explicitly asking it to.
$7045 | select * | Export-Csv -Path test.csv -NoTypeInformation
$7045 | Export-Csv -Path test.csv -NoTypeInformation
$7045 | Select-Object Timestamp, Id, 'Machine Name', 'Service Name', 'Image Path', 'RunAsUser', 'Installed By', Suspicious | Export-Csv -Path test.csv -NoTypeInformation
I read the Export-CSV documentation on MS. Searched StackOverFlow for some tips, I think it has something to do with PS checking the first Row and then compares if the property exists for the second row and so on.
Thank you
The issue you're experiencing is partially because of how objects are displayed to the console, the first object's Properties determines the displayed Properties (Columns) to the console.
The bigger problem though, is that Export-Csv will not export those properties that do not match with first object's properties unless they're explicitly added to the remaining objects or the objects are reconstructed, for this one easy way is to use Select-Object as you have pointed out in the question.
Given the following example:
$test = #(
[pscustomobject]#{
A = 'ValA'
}
[pscustomobject]#{
A = 'ValA'
B = 'ValB'
}
[pscustomobject]#{
C = 'ValC'
D = 'ValD'
E = 'ValE'
}
)
Format-Table will not display the properties B to E:
$test | Format-Table
A
-
ValA
ValA
Format-List can display the objects properly, this is because each property with it's corresponding value has it's own console line in the display:
PS /> $test | Format-List
A : ValA
A : ValA
B : ValB
C : ValC
D : ValD
E : ValE
Export-Csv and ConvertTo-Csv will also miss properties B to E:
$test | ConvertTo-Csv
"A"
"ValA"
"ValA"
You have different options as a workaround for this, you could either add the Suspicious property to all objects and for those events that are not suspicious you could add $null as Value.
Another workaround is to use Select-Object explicitly calling the Suspicious property (this works because you know the property is there and you know it's Name).
If you did not know how many properties your objects had, a dynamic way to solve this would be to discover their properties using the PSObject intrinsic member.
using namespace System.Collections.Generic
function ConvertTo-NormalizedObject {
[CmdletBinding()]
param(
[Parameter(ValueFromPipeline, Mandatory)]
[object[]] $InputObject
)
begin {
$list = [List[object]]::new()
$props = [HashSet[string]]::new([StringComparer]::InvariantCultureIgnoreCase)
}
process {
foreach($object in $InputObject) {
$list.Add($object)
foreach($property in $object.PSObject.Properties) {
$null = $props.Add($property.Name)
}
}
}
end {
$list | Select-Object ([object[]] $props)
}
}
Usage:
# From Pipeline
$test | ConvertTo-NormalizedObject | Format-Table
# From Positional / Named parameter binding
ConvertTo-NormalizedObject $test | Format-Table
Lastly, a pretty easy way of doing it thanks to Select-Object -Unique:
$prop = $test.ForEach{ $_.PSObject.Properties.Name } | Select-Object -Unique
$test | Select-Object $prop
Using $test for this example, the result would become:
A B C D E
- - - - -
ValA
ValA ValB
ValC ValD ValE
Continuing from my previous answer, you can add a column Suspicious straight away if you take out the Where-Object filter and simply add another calculated property to the Select-Object cmdlet:
# create a regex for the suspicious executables:
$sus = '(powershell|cmd|psexesvc)\.exe'
# alternatively you can join the array items like this:
# $sus = ('powershell.exe','cmd.exe','psexesvc.exe' | ForEach-Object {[regex]::Escape($_)}) -join '|'
$7045 = Get-WinEvent -FilterHashtable #{ LogName = 'System';Id = 7045 } |
Select-Object Id,
#{N='Timestamp';E={$_.TimeCreated.ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:ssZ')}},
#{N='Machine Name';E={$_.MachineName}},
#{N='Service Name'; E={$_.Properties[0].Value}},
#{N='Image Path'; E={$_.Properties[1].Value}},
#{N='RunAsUser'; E={$_.Properties[4].Value}},
#{N='Installed By'; E={$_.UserId}},
#{N='Suspicious'; E={
if ($_.Properties[1].Value -match $sus) { 'Yes' } else {'No'}
}}
$7045 | Export-Csv -Path 'X:\Services.csv' -UseCulture -NoTypeInformation
Because you have many columns, this will not fit the console width anymore if you do $7045 | Format-Table, but the CSV file will hold all columns you wanted.
I added switch -UseCulture to the Export-Csv cmdlet, which makes sure you can simply double-click the csv file so it opens correctly in your Excel.
As sidenote: Please do not use those curly so-called 'smart-quotes' in code as they may lead to unforeseen errors. Straighten these ’ thingies and use normal double or single quotes (" and ')
As part of a CI pipeline, I need to generate a list of all the files in the repository, including certain properties of them, and to Out-File it into a file. The command I use:
Get-ChildItem $Path -File -Recurse | Select-Object -Property LastWriteTime, #
{
label = "Size(KB)"
expr = { [string]::Format("{0:0.00}", $_.Length/1KB) }
}, FullName, <some_other_property> | Out-File $OutputFile
My problem is, that running this script from the command line gives the desired result.
However, running this during a Azure Pipeline build does 2 bad things:
cuts the lines in the FullName column when they are long:
LastWriteTime Size(KB) Name
------------ ------- ----
<some_date> <some size> ASomeWhatLong...
Doesn't display the rest of the properties such as <some_other_property>
If I turn FullName into Name it all goes OK, but I really need the FullName property.
Because I'm working in a Air-gapped environment I can't copy all the outputs and everything.
I've tried using the -Width flag for Out-File with no success.
I believe what's happening under the hood, ps uses the ToString() method of the object created which outputs it as the Format-Table cmdlet does. You get truncated properties because of the Window's size. To look at it you could use:
(Get-Host).ui.RawUI.WindowSize
Probably this is too small.
What I would suggest is the following:
Pipe the object into Format-Table
Get-ChildItem $Path -File -Recurse | Select-Object -Property LastWriteTime, #
{
label = "Size(KB)"
expr = { [string]::Format("{0:0.00}", $_.Length/1KB) }
}, FullName, <some_other_property> | Format-Table | Out-String | Out-File $OutputFile
This probably won't work as it is, but you could play with the Format-Table's properties like: -Wrap. By default it will allocate enough space for the first properties, and the last one it would try to 'fit' it, which might look as:
LastWriteTime Size(KB) Name
------------ ------- ----
<some_date> <some size> ASomeWhatLong
foobarfoobarfo
foobarfoobarfo
To solve this, you can use the -Property argument, which needs to be as:
$propertWidth = [int]((Get-Host).ui.RawUI.WindowSize.Width / 3)
$property = #(
#{ Expression = 'LastWriteTime'; Width = $propertWidth; },
#{ Expression = 'Size(KB)'; Width = $propertWidth; },
#{ Expression = 'FullName'; Width = $propertWidth; }
)
... | Format-Table -Property $property -Wrap | ...
If you don't mind having a JSON into your file, you could use:
Get-ChildItem $Path -File -Recurse | Select-Object -Property LastWriteTime, #
{
label = "Size(KB)"
expr = { [string]::Format("{0:0.00}", $_.Length/1KB) }
}, FullName, <some_other_property> | ConvertTo-Json | Out-File $OutputFile
But take into account that ConvertTo-Json has a default Depth of 2. This will also truncate your objects if you have nested objects. But as far as property lengths, it will do fine.
I have deleted my original question because I believe I have a more efficient way to run my script, thus I'm changing my question.
$scrubFileOneDelim = "|"
$scrubFileTwoDelim = "|"
$scrubFileOneBal = 2
$scrubFileTwoBal = 56
$scrubFileOneAcctNum = 0
$scrubFileTwoAcctNum = 0
$ColumnsF1 = Get-Content $scrubFileOne | ForEach-Object{($_.split($scrubFileOneDelim)).Count} | Measure-Object -Maximum | Select-Object -ExpandProperty Maximum
$ColumnsF2 = Get-Content $scrubFileTwo | ForEach-Object{($_.split($scrubFileTwoDelim)).Count} | Measure-Object -Maximum | Select-Object -ExpandProperty Maximum
$useColumnsF1 = $ColumnsF1-1;
$useColumnsF2 = $ColumnsF2-1;
$fileOne = import-csv "$scrubFileOne" -Delimiter "$scrubFileOneDelim" -Header (0..$useColumnsF1) | select -Property #{label="BALANCE";expression={$($_.$scrubFileOneBal)}},#{label="ACCTNUM";expression={$($_.$scrubFileOneAcctNum)}}
$fileTwo = import-csv "$scrubFileTwo" -Delimiter "$scrubFileTwoDelim" -Header (0..$useColumnsF2) | select -Property #{label="BALANCE";expression={$($_.$scrubFileTwoBal)}},#{label="ACCTNUM";expression={$($_.$scrubFileTwoAcctNum)}}
$hash = #{}
$hashTwo = #{}
$fileOne | foreach { $hash.add($_.ACCTNUM, $_.BALANCE) }
$fileTwo | foreach { $hashTwo.add($_.ACCTNUM, $_.BALANCE) }
In this script I'm doing the following, counting header's to return the count and use it in a range operator in order to dynamically insert headers for later manipulation. Then I'm importing 2 CSV files. I'm taking those CSV files and pushing them into their own hashtable.
Just for an idea of what I'm trying to do from here...
CSV1 (as a hashtable) looks like this:
Name Value
---- -----
000000000001 000000285+
000000000002 000031000+
000000000003 000004685+
000000000004 000025877+
000000000005 000000001+
000000000006 000031000+
000000000007 000018137+
000000000008 000000000+
CSV2 (as a hashtable) looks like this:
Name Value
---- -----
000000000001 000008411+
000000000003 000018137+
000000000007 000042865+
000000000008 000009761+
I would like to create a third hash table. It will have all the "NAME" items from CSV2, but I don't want the "VALUE" from CSV2, I want it to have the "VALUE"s that CSV1 has. So in the end result would look like this.
Name Value
---- -----
000000000001 000000285+
000000000003 000004685+
000000000007 000018137+
000000000008 000000000+
Ultimately I want this to be exported as a csv.
I have tried this with just doing a compare-object, not doing the hashtables with the following code, but I abandoned trying to do it this way because file 1 may have 100,000 "accounts" where file 2 only has 200, and the result I was getting listed close to the 100,000 accounts that I didn't want to be in the result. They had the right balances but I want a file that only has those balances for the accounts listed in file 2. This code below isn't really a part of my question, just showing something I've tried. I just think this is much easier and faster with a hash table now so I would like to go that route.
#Find and Rename the BALANCE and ACCOUNT NUMBER columns in both files.
$fileOne = import-csv "$scrubFileOne" -Delimiter "$scrubFileOneDelim" -Header (0..$useColumnsF1) | select -Property #{label="BALANCE";expression={$($_.$scrubFileOneBal)}},#{label="ACCT-NUM";expression={$($_.$scrubFileOneAcctNum)}}
$fileTwo = import-csv "$scrubFileTwo" -Delimiter "$scrubFileTwoDelim" -Header (0..$useColumnsF2) | select -Property #{label="BALANCE";expression={$($_.$scrubFileTwoBal)}},#{label="ACCT-NUM";expression={$($_.$scrubFileTwoAcctNum)}}
Compare-Object $fileOne $fileTwo -Property 'BALANCE','ACCTNUM' -IncludeEqual -PassThru | Where-Object{$_.sideIndicator -eq "<="} | select * -Exclude SideIndicator | export-csv -notype "C:\test\f1.txt"
What you are after is filtering the Compare-Object function. This will show only one side of the result. YOu will need to place this before you exclude that property for it to work.
| Where-Object{$_.sideIndicator -eq "<="} |
Assuming that you have the following hash tables:
$hash = #{
'000000000001' = '000000285+';
'000000000002' = '000031000+';
'000000000003' = '000004685+';
'000000000004' = '000025877+';
'000000000005' = '000000001+';
'000000000006' = '000031000+';
'000000000007' = '000018137+';
'000000000008' = '000000000+';
}
$hashTwo = #{
'000000000001' = '000008411+';
'000000000003' = '000018137+';
'000000000007' = '000042865+';
'000000000008' = '000009761+';
}
you can create the third hash table by iterating over the keys from the second hash table and then assigning those keys to the value from the first hash table.
$hashThree = #{}
ForEach ($key In $hashTwo.Keys) {
$hashThree["$key"] = $hash["$key"]
}
$hashThree
The output of $hashThree is:
Name Value
---- -----
000000000007 000018137+
000000000001 000000285+
000000000008 000000000+
000000000003 000004685+
If you want the order of the data maintained (and you are using PowerShell 6 Core), you can use [ordered]#{} when creating the hash tables.
I'm trying to take an array of PSObjects similar to
#{BakId=27; Name=DB_A; Lsn=123; File=A_01; Size=987}
#{BakId=28; Name=DB_B; Lsn=456; File=B_01; Size=876}
#{BakId=28; Name=DB_B; Lsn=456; File=B_02; Size=765}
#{BakId=28; Name=DB_B; Lsn=456; File=B_03; Size=654}
And create a new grouped object that removes redundant header info.
BakId Lsn Name Files
27 123 DB_A {#{File=A_01.bak;Size=777}}
28 456 DB_B {#{File=B_01.bak;Size=888}, #{File=B_02.bak;Size=999}, ...}
I tried using group-object but can only get it to work for one property. (all grouped properties go into Group.Name as a a string of comma separated values.)
This is the best I've come up with, but feels hacky.
$list | Group-Object -Property BakId | % {
$BakId = $_.Name
$Lsn = $_.Group[0].Lsn # <--- is there a better way than this?
$Name = $_.Group[0].Name # <--- Ditto
$Files = $_.Group | Select-Object -Property SequenceNumber, Size
Write-Output (New-Object -TypeName PSObject -Property #{ BakId=$BakId;Files = $Files })
}
Is there a better way?
Thanks
You can simplify the approach to constructing the output objects by using a single Select-Object call with calculated properties, and, relying on the order of the grouping properties, access their group-specific values via the .Values collection:
$list | Group-Object -Property BakId, Lsn, Name |
Select-Object #{n='BakId'; e={ $_.Values[0] }},
#{n='Lsn'; e={ $_.Values[1] }},
#{n='Name'; e={ $_.Values[2] }},
#{n='Files'; e={ $_.Group | Select-Object File, Size }}
Note:
$_.Values[<ndx>] takes the place of $_.Group[0].<name> in your approach; note that the latter only makes sense for the actual grouping properties, because any others will not be uniform throughout the group.
The .Values collection ([System.Collections.ArrayList]) on each group object output from Group-Object ([Microsoft.PowerShell.Commands.GroupInfo] instance) contains that group's shared grouping-property values as-is (original type), in the order specified.
By contrast, property .Name contains the stringified combination of all grouping-property values: a string containing a comma-separated list.
Unfortunately, such details are currently missing from Get-Help Group-Object.
If you wanted to avoid having to repeat the grouping properties (e.g., in preparation for writing a function wrapper; PSv3+):
$props = 'BakId', 'Lsn', 'Name'
$list | Group-Object -Property $props | ForEach-Object {
$propDefs = [ordered] #{}
foreach ($i in 0..($props.Length-1)) { $propDefs.[$props[$i]] = $_.Values[$i] }
$propDefs.Files = $_.Group | Select-Object File, Size
New-Object PSCustomObject -Property $propDefs
}