I have seen powershell script which also I have in mind. What I would like to add though is another column which would show the side indicator comparators ("==", "<=", "=>") and be named them as MATCH(if "==") and MISMATCH(if "<=" and "=>").
Any advise on how I would do this?
Here is the link of the script (Credits to Florent Courtay)
How can i reorganise powershell's compare-object output?
$a = Compare-Object (Import-Csv 'C:\temp\f1.csv') (Import-Csv 'C:\temp\f2.csv') -property Header,Value
$a | Group-Object -Property Header | % { New-Object -TypeName psobject -Property #{Header=$_.name;newValue=$_.group[0].Value;oldValue=$_.group[1].Value}}
========================================================================
The output I have in mind:
Header1 Old Value New Value STATUS
------ --------- --------- -----------
String1 Value 1 Value 2 MATCH
String2 Value 3 Value 4 MATCH
String3 NA Value 5 MISMATCH
String4 Value 6 NA MISMATCH
Here's a self-contained solution; simply replace the ConvertFrom-Csv calls with your Import-Csv calls:
# Sample CSV input.
$csv1 = #'
Header,Value
a,1
b,2
c,3
'#
$csv2 = #'
Header,Value
a,1a
b,2
d,4
'#
Compare-Object (ConvertFrom-Csv $csv1) (ConvertFrom-Csv $csv2) -Property Header, Value |
Group-Object Header | Sort-Object Name | ForEach-Object {
$newValIndex, $oldValIndex = ((1, 0), (0, 1))[$_.Group[0].SideIndicator -eq '=>']
[pscustomobject] #{
Header = $_.Name
OldValue = ('NA', $_.Group[$oldValIndex].Value)[$null -ne $_.Group[$oldValIndex].Value]
NewValue = ('NA', $_.Group[$newValIndex].Value)[$null -ne $_.Group[$newValIndex].Value]
Status = ('MISMATCH', 'MATCH')[$_.Group.Count -gt 1]
}
}
The above yields:
Header OldValue NewValue Status
------ -------- -------- ------
a 1 1a MATCH
c 3 NA MISMATCH
d NA 4 MISMATCH
Note:
The assumption is that a given Header column value appears at most once in each input file.
The Sort-Object Name call is needed to sort the output by Header valuesThanks, LotPings.
, because, due to how Compare-Object orders its output (right-side-only items first), the order of groups created by Group-Object would not automatically reflect the 1st CSV's order of header values (d would appear before c).
Related
I am trying to get a list of files and a count of the number of rows in each file displayed in a table consisting of two columns, Name and Lines.
I have tried using format table but I don't think the problem is with the format of the table and more to do with my results being separate results. See below
#Get a list of files in the filepath location
$files = Get-ChildItem $filepath
$files | ForEach-Object { $_ ; $_ | Get-Content | Measure-Object -Line} | Format-Table Name,Lines
Expected results
Name Lines
File A
9
File B
89
Actual Results
Name Lines
File A
9
File B
89
Another approach how to make a custom object like this: Using PowerShell's Calculated Properties:
$files | Select-Object -Property #{ N = 'Name' ; E = { $_.Name} },
#{ N = 'Lines'; E = { ($_ | Get-Content | Measure-Object -Line).Lines } }
Name Lines
---- -----
dotNetEnumClass.ps1 232
DotNetVersions.ps1 9
dotNETversionTable.ps1 64
Typically you would make a custom object like this, instead of outputting two different kinds of objects.
$files | ForEach-Object {
$lines = $_ | Get-Content | Measure-Object -Line
[pscustomobject]#{name = $_.name
lines = $lines.lines}
}
name lines
---- -----
rof.ps1 11
rof.ps1~ 7
wai.ps1 2
wai.ps1~ 1
I have deleted my original question because I believe I have a more efficient way to run my script, thus I'm changing my question.
$scrubFileOneDelim = "|"
$scrubFileTwoDelim = "|"
$scrubFileOneBal = 2
$scrubFileTwoBal = 56
$scrubFileOneAcctNum = 0
$scrubFileTwoAcctNum = 0
$ColumnsF1 = Get-Content $scrubFileOne | ForEach-Object{($_.split($scrubFileOneDelim)).Count} | Measure-Object -Maximum | Select-Object -ExpandProperty Maximum
$ColumnsF2 = Get-Content $scrubFileTwo | ForEach-Object{($_.split($scrubFileTwoDelim)).Count} | Measure-Object -Maximum | Select-Object -ExpandProperty Maximum
$useColumnsF1 = $ColumnsF1-1;
$useColumnsF2 = $ColumnsF2-1;
$fileOne = import-csv "$scrubFileOne" -Delimiter "$scrubFileOneDelim" -Header (0..$useColumnsF1) | select -Property #{label="BALANCE";expression={$($_.$scrubFileOneBal)}},#{label="ACCTNUM";expression={$($_.$scrubFileOneAcctNum)}}
$fileTwo = import-csv "$scrubFileTwo" -Delimiter "$scrubFileTwoDelim" -Header (0..$useColumnsF2) | select -Property #{label="BALANCE";expression={$($_.$scrubFileTwoBal)}},#{label="ACCTNUM";expression={$($_.$scrubFileTwoAcctNum)}}
$hash = #{}
$hashTwo = #{}
$fileOne | foreach { $hash.add($_.ACCTNUM, $_.BALANCE) }
$fileTwo | foreach { $hashTwo.add($_.ACCTNUM, $_.BALANCE) }
In this script I'm doing the following, counting header's to return the count and use it in a range operator in order to dynamically insert headers for later manipulation. Then I'm importing 2 CSV files. I'm taking those CSV files and pushing them into their own hashtable.
Just for an idea of what I'm trying to do from here...
CSV1 (as a hashtable) looks like this:
Name Value
---- -----
000000000001 000000285+
000000000002 000031000+
000000000003 000004685+
000000000004 000025877+
000000000005 000000001+
000000000006 000031000+
000000000007 000018137+
000000000008 000000000+
CSV2 (as a hashtable) looks like this:
Name Value
---- -----
000000000001 000008411+
000000000003 000018137+
000000000007 000042865+
000000000008 000009761+
I would like to create a third hash table. It will have all the "NAME" items from CSV2, but I don't want the "VALUE" from CSV2, I want it to have the "VALUE"s that CSV1 has. So in the end result would look like this.
Name Value
---- -----
000000000001 000000285+
000000000003 000004685+
000000000007 000018137+
000000000008 000000000+
Ultimately I want this to be exported as a csv.
I have tried this with just doing a compare-object, not doing the hashtables with the following code, but I abandoned trying to do it this way because file 1 may have 100,000 "accounts" where file 2 only has 200, and the result I was getting listed close to the 100,000 accounts that I didn't want to be in the result. They had the right balances but I want a file that only has those balances for the accounts listed in file 2. This code below isn't really a part of my question, just showing something I've tried. I just think this is much easier and faster with a hash table now so I would like to go that route.
#Find and Rename the BALANCE and ACCOUNT NUMBER columns in both files.
$fileOne = import-csv "$scrubFileOne" -Delimiter "$scrubFileOneDelim" -Header (0..$useColumnsF1) | select -Property #{label="BALANCE";expression={$($_.$scrubFileOneBal)}},#{label="ACCT-NUM";expression={$($_.$scrubFileOneAcctNum)}}
$fileTwo = import-csv "$scrubFileTwo" -Delimiter "$scrubFileTwoDelim" -Header (0..$useColumnsF2) | select -Property #{label="BALANCE";expression={$($_.$scrubFileTwoBal)}},#{label="ACCT-NUM";expression={$($_.$scrubFileTwoAcctNum)}}
Compare-Object $fileOne $fileTwo -Property 'BALANCE','ACCTNUM' -IncludeEqual -PassThru | Where-Object{$_.sideIndicator -eq "<="} | select * -Exclude SideIndicator | export-csv -notype "C:\test\f1.txt"
What you are after is filtering the Compare-Object function. This will show only one side of the result. YOu will need to place this before you exclude that property for it to work.
| Where-Object{$_.sideIndicator -eq "<="} |
Assuming that you have the following hash tables:
$hash = #{
'000000000001' = '000000285+';
'000000000002' = '000031000+';
'000000000003' = '000004685+';
'000000000004' = '000025877+';
'000000000005' = '000000001+';
'000000000006' = '000031000+';
'000000000007' = '000018137+';
'000000000008' = '000000000+';
}
$hashTwo = #{
'000000000001' = '000008411+';
'000000000003' = '000018137+';
'000000000007' = '000042865+';
'000000000008' = '000009761+';
}
you can create the third hash table by iterating over the keys from the second hash table and then assigning those keys to the value from the first hash table.
$hashThree = #{}
ForEach ($key In $hashTwo.Keys) {
$hashThree["$key"] = $hash["$key"]
}
$hashThree
The output of $hashThree is:
Name Value
---- -----
000000000007 000018137+
000000000001 000000285+
000000000008 000000000+
000000000003 000004685+
If you want the order of the data maintained (and you are using PowerShell 6 Core), you can use [ordered]#{} when creating the hash tables.
I have two CSV Files which look like this:
test.csv:
"Col1","Col2"
"1111","1"
"1122","2"
"1111","3"
"1121","2"
"1121","2"
"1133","2"
"1133","2"
The second looks like this:
test2.csv:
"Number","signs"
"1111","ABC"
"1122","DEF"
"1111","ABC"
"1121","ABC"
"1133","GHI"
Now the goal is to get a summary of all points from test.csv assigned to the "signs" of test2.csv. Reference are the numbers, as you may see.
Should be something like this:
ABC = 8
DEF = 2
GHI = 4
I have tried to test this out but cannot get the goal. What I have so far is:
$var = "C:\PathToCSV"
$csv1 = Import-Csv "$var\test.csv"
$csv2 = Import-Csv "$var\test2.csv"
# Process: group by 'Item' then sum 'Average' for each group
# and create output objects on the fly
$test1 = $csv1 | Group-Object Col1 | ForEach-Object {
New-Object psobject -Property #{
Col1 = $_.Name
Sum = ($_.Group | Measure-Object Col2 -Sum).Sum
}
}
But this gives me back the following output:
Ps> $test1
Sum Col1
--- ----
4 1111
2 1122
4 1121
4 1133
I am not able to get the summary and the mapping of the signs.
Not sure if I understand your question correctly, but I'm going to assume that for each value from the column "signs" you want to lookup the values from the column "Number" in the second CSV and then calculate the sum of the column "Col2" for all matches.
For that I'd build a hashtable with the pre-calculated sums for the unique values from "Col1":
$h1 = #{}
$csv1 | ForEach-Object {
$h1[$_.Col1] += [int]$_.Col2
}
and then build a second hashtable to sum up the lookup results for the values from the second CSV:
$h2 = #{}
$csv2 | ForEach-Object {
$h2[$_.signs] += $h1[$_.Number]
}
However, that produced a different value for "ABC" than what you stated as the desired result in your question when I processed your sample data:
Name Value
---- -----
ABC 12
GHI 4
DEF 2
Or did you mean you want to sum up the corresponding values for the unique numbers for each sign? For that you'd change the second code snippet to something like this:
$h2 = #{}
$csv2 | Group-Object signs | ForEach-Object {
$name = $_.Name
$_.Group | Select-Object -Unique -Expand Number | ForEach-Object {
$h2[$name] += $h1[$_]
}
}
That would produce the desired result from your question:
Name Value
---- -----
ABC 8
GHI 4
DEF 2
I have two CSV files I need to compare. Both of which may (or may not) contain an additional delimited field (delimited via a "|" character), like so:
(new.csv)
Title,Key,Value
Jason,Son,Hair=Red|Eyes=Blue
James,Son,Hair=Brown|Eyes=Green
Ron,Father,Hair=Black
Susan,Mother,Hair=Black|Eyes=Brown|Dress=Green
(old.csv)
Title,Key,Value
Jason,Son,Hair=Purple|Eyes=Blue
James,Son,Hair=Brown|Eyes=Green
Ron,Father,Hair=Purple
Susan,Mother,Hair=Black|Eyes=Brown|Dress=Blue
My problem comes in when I attempt to compare the two files...
$fileNew = "new.csv"
$fileOld = "old.csv"
$fileDiffOutputFile = "diff.txt"
$csvNewLog = (Import-CSV ($fileNew))
$csvOldLog = (Import-CSV ($fileOld))
$varDifferences = Compare-Object $csvOldLog $csvNewLog -property Title,Value
$varDifferences | Group-Object -Property Title | % {New-Object -TypeName PSObject -Property #{ NewValue=($_.group[0].Value); Title=$_.name; OldValue=($_.group[1].Value) } } | Out-File $fileDiffOutputFile -Append
Resulting in this output:
(diff.txt)
OldValue Title NewValue
-------- ----- --------
Hair=Purple|Eyes=Blue Jason Hair=Red|Eyes=Blue
Hair=Purple Ron Hair=Black
Hair=Black|Eyes=Brown|D... Susan Hair=Black|Eyes=Brown|...
Some of the values are inevitably going to extend out past the max length of the column, as it does with Susan above.
So, my question could have a couple of solutions that I can think of:
Is there an easier way to isolate the values so that I only pull out the changed values, and not the entire string of delimited values?
If not, is it possible to get the format to show the entire string (including the unchanged values part of the delimited string) instead?
If you include a format-table -wrap in your last line, like so?
$fileNew = "new.csv"
$fileOld = "old.csv"
$fileDiffOutputFile = "diff.txt"
$csvNewLog = (Import-CSV ($fileNew))
$csvOldLog = (Import-CSV ($fileOld))
$varDifferences = Compare-Object $csvOldLog $csvNewLog -property Title,Value
$varDifferences | Group-Object -Property Title | % {New-Object -TypeName PSObject -Property #{ NewValue=($_.group[0].Value); Title=$_.name; OldValue=($_.group[1].Value) } } | Format-Table -wrap | Out-File $fileDiffOutputFile -Append
So I have a CSV file which I need to manipulate a bit, select the data I need and export to another CSV file.
The code I have is:
$rawCSV = "C:\Files\raw.csv"
$outputCSV = "C:\Files\output.csv"
Import-Csv -Header #("a","b","c","d") -Path $rawCSV |
select -Skip 7 |
Where-Object { $_.b.length -gt 1 } |
ft b,a,c,d |
Out-File $outputCSV
So this code uses the Import-Csv command to allow me to select just the columns I need, add some headers in the order I want and then I am simply putting the output in to a CSV file called $outputCSV. The contents of this output file look something like this:
b a c d
- - - -
john smith 29 England
mary poopins 79 Walton
I am not sure what the delimiter is in this output and rather than these columns being treated as individuals, they are treated as just one column. I have gone on further to replace all the spaces with a comma using the code:
$b = foreach ($line in $a)
{
$fields = $line -split '`n'
foreach ($field in $fields)
{
$field -replace " +",","
}
}
Which produces a file that looks like this:
b,a,c,d
john,smith,29,England
mary,poppins,79,Walton
But these are all still treated as one column instead of four separate columns as I need.
* UPDATE *
Using the answer given by #, I now get a file looking like this:
Don't use ft to reorder your columns - it's intended to format output for the screen, not really suitable for CSV.
"Manual" solution:
$rawCSV = "C:\Files\raw.csv"
$outputCSV = "C:\Files\output.csv"
# Import and filter your raw data
$RawData = Import-Csv -Header #("a","b","c","d") -Path $rawCSV
$Data = $RawData | Select -Skip 7 | Where-Object { $_.b.length -gt 1 }
# Write your headers to the output file
"b","a","c","d" -join ',' | Out-File $outputCSV -Force
$ReorderedData = foreach($Row in $Data){
# Reorder the columns in each row
'{0},{1},{2},{3}' -f $Row.b , $Row.a , $Row.c, $Row.d
}
# Write the reordered rows to the output file
$ReorderedData | Out-File $outputCSV -Append
Using Export-Csv:
As of PowerShell 3.0, you could also push the rows into a [pscustomobject] and pipe that to Export-Csv (pscustomobject preserves the order in which you supply the properties):
$rawCSV = "C:\Files\raw.csv"
$outputCSV = "C:\Files\output.csv"
# Import and filter your raw data
$RawData = Import-Csv -Header #("a","b","c","d") -Path $rawCSV
$Data = $RawData | Select -Skip 7 | Where-Object { $_.b.length -gt 1 }
# Take the columns you're interested in, put them into new custom objects and export to CSV
$Data | ForEach-Object {
[pscustomobject]#{ "b" = $_.b; "a" = $_.a; "c" = $_.c; "d" = $_.d }
} | Export-Csv -NoTypeInformation $outputCSV
Export-Csv will take care of enclosing strings in quotes to escape ',' properly (one thing less for you to worry about)
First of all, what your raw CSV file looks like? If it's already like this
john,smith,29,England
mary,poppins,79,Walton
then import-csv will give you an array of objects which you can easily manipulate (and objects are the main reason to use PowerShell ;). For example, to check what you have after import:
$r = Import-Csv -Path $rawCSV -Header #("b","a","c","d")
$r.GetType()
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True Object[] System.Array
$r[0] | get-member
TypeName: System.Management.Automation.PSCustomObject
Name MemberType Definition
---- ---------- ----------
Equals Method bool Equals(System.Object obj)
GetHashCode Method int GetHashCode()
GetType Method type GetType()
ToString Method string ToString()
a NoteProperty System.String a=smith
b NoteProperty System.String b=john
c NoteProperty System.String c=29
d NoteProperty System.String d=England
For now you have array of objects with properties named "a","b","c","d". To manipulate objects you have select-object cmdlet:
$r | Select-Object a,b,c,d
a b c d
- - - -
smith john 29 England
poppins mary 79 Walton
And after all use export-csv to set the output file:
$r | where { $_.b.length -gt 1 } |
select a,b,c,d |
Export-Csv -NoTypeInformation -Encoding utf8 -path $outputCSV
I could think of two possible reasons why your data teated as one column:
consuming application expect different encoding and can't find
delimiters
delimiters are not commas but something else