Add up the data if the reference from another file is correct - powershell

I have two CSV Files which look like this:
test.csv:
"Col1","Col2"
"1111","1"
"1122","2"
"1111","3"
"1121","2"
"1121","2"
"1133","2"
"1133","2"
The second looks like this:
test2.csv:
"Number","signs"
"1111","ABC"
"1122","DEF"
"1111","ABC"
"1121","ABC"
"1133","GHI"
Now the goal is to get a summary of all points from test.csv assigned to the "signs" of test2.csv. Reference are the numbers, as you may see.
Should be something like this:
ABC = 8
DEF = 2
GHI = 4
I have tried to test this out but cannot get the goal. What I have so far is:
$var = "C:\PathToCSV"
$csv1 = Import-Csv "$var\test.csv"
$csv2 = Import-Csv "$var\test2.csv"
# Process: group by 'Item' then sum 'Average' for each group
# and create output objects on the fly
$test1 = $csv1 | Group-Object Col1 | ForEach-Object {
New-Object psobject -Property #{
Col1 = $_.Name
Sum = ($_.Group | Measure-Object Col2 -Sum).Sum
}
}
But this gives me back the following output:
Ps> $test1
Sum Col1
--- ----
4 1111
2 1122
4 1121
4 1133
I am not able to get the summary and the mapping of the signs.

Not sure if I understand your question correctly, but I'm going to assume that for each value from the column "signs" you want to lookup the values from the column "Number" in the second CSV and then calculate the sum of the column "Col2" for all matches.
For that I'd build a hashtable with the pre-calculated sums for the unique values from "Col1":
$h1 = #{}
$csv1 | ForEach-Object {
$h1[$_.Col1] += [int]$_.Col2
}
and then build a second hashtable to sum up the lookup results for the values from the second CSV:
$h2 = #{}
$csv2 | ForEach-Object {
$h2[$_.signs] += $h1[$_.Number]
}
However, that produced a different value for "ABC" than what you stated as the desired result in your question when I processed your sample data:
Name Value
---- -----
ABC 12
GHI 4
DEF 2
Or did you mean you want to sum up the corresponding values for the unique numbers for each sign? For that you'd change the second code snippet to something like this:
$h2 = #{}
$csv2 | Group-Object signs | ForEach-Object {
$name = $_.Name
$_.Group | Select-Object -Unique -Expand Number | ForEach-Object {
$h2[$name] += $h1[$_]
}
}
That would produce the desired result from your question:
Name Value
---- -----
ABC 8
GHI 4
DEF 2

Related

Get results of For-Each arrays and display in a table with column headers one line per results

I am trying to get a list of files and a count of the number of rows in each file displayed in a table consisting of two columns, Name and Lines.
I have tried using format table but I don't think the problem is with the format of the table and more to do with my results being separate results. See below
#Get a list of files in the filepath location
$files = Get-ChildItem $filepath
$files | ForEach-Object { $_ ; $_ | Get-Content | Measure-Object -Line} | Format-Table Name,Lines
Expected results
Name Lines
File A
9
File B
89
Actual Results
Name Lines
File A
9
File B
89
Another approach how to make a custom object like this: Using PowerShell's Calculated Properties:
$files | Select-Object -Property #{ N = 'Name' ; E = { $_.Name} },
#{ N = 'Lines'; E = { ($_ | Get-Content | Measure-Object -Line).Lines } }
Name Lines
---- -----
dotNetEnumClass.ps1 232
DotNetVersions.ps1 9
dotNETversionTable.ps1 64
Typically you would make a custom object like this, instead of outputting two different kinds of objects.
$files | ForEach-Object {
$lines = $_ | Get-Content | Measure-Object -Line
[pscustomobject]#{name = $_.name
lines = $lines.lines}
}
name lines
---- -----
rof.ps1 11
rof.ps1~ 7
wai.ps1 2
wai.ps1~ 1

Powershell Compare-object IF different then ONLY list items from one file, not both

I have deleted my original question because I believe I have a more efficient way to run my script, thus I'm changing my question.
$scrubFileOneDelim = "|"
$scrubFileTwoDelim = "|"
$scrubFileOneBal = 2
$scrubFileTwoBal = 56
$scrubFileOneAcctNum = 0
$scrubFileTwoAcctNum = 0
$ColumnsF1 = Get-Content $scrubFileOne | ForEach-Object{($_.split($scrubFileOneDelim)).Count} | Measure-Object -Maximum | Select-Object -ExpandProperty Maximum
$ColumnsF2 = Get-Content $scrubFileTwo | ForEach-Object{($_.split($scrubFileTwoDelim)).Count} | Measure-Object -Maximum | Select-Object -ExpandProperty Maximum
$useColumnsF1 = $ColumnsF1-1;
$useColumnsF2 = $ColumnsF2-1;
$fileOne = import-csv "$scrubFileOne" -Delimiter "$scrubFileOneDelim" -Header (0..$useColumnsF1) | select -Property #{label="BALANCE";expression={$($_.$scrubFileOneBal)}},#{label="ACCTNUM";expression={$($_.$scrubFileOneAcctNum)}}
$fileTwo = import-csv "$scrubFileTwo" -Delimiter "$scrubFileTwoDelim" -Header (0..$useColumnsF2) | select -Property #{label="BALANCE";expression={$($_.$scrubFileTwoBal)}},#{label="ACCTNUM";expression={$($_.$scrubFileTwoAcctNum)}}
$hash = #{}
$hashTwo = #{}
$fileOne | foreach { $hash.add($_.ACCTNUM, $_.BALANCE) }
$fileTwo | foreach { $hashTwo.add($_.ACCTNUM, $_.BALANCE) }
In this script I'm doing the following, counting header's to return the count and use it in a range operator in order to dynamically insert headers for later manipulation. Then I'm importing 2 CSV files. I'm taking those CSV files and pushing them into their own hashtable.
Just for an idea of what I'm trying to do from here...
CSV1 (as a hashtable) looks like this:
Name Value
---- -----
000000000001 000000285+
000000000002 000031000+
000000000003 000004685+
000000000004 000025877+
000000000005 000000001+
000000000006 000031000+
000000000007 000018137+
000000000008 000000000+
CSV2 (as a hashtable) looks like this:
Name Value
---- -----
000000000001 000008411+
000000000003 000018137+
000000000007 000042865+
000000000008 000009761+
I would like to create a third hash table. It will have all the "NAME" items from CSV2, but I don't want the "VALUE" from CSV2, I want it to have the "VALUE"s that CSV1 has. So in the end result would look like this.
Name Value
---- -----
000000000001 000000285+
000000000003 000004685+
000000000007 000018137+
000000000008 000000000+
Ultimately I want this to be exported as a csv.
I have tried this with just doing a compare-object, not doing the hashtables with the following code, but I abandoned trying to do it this way because file 1 may have 100,000 "accounts" where file 2 only has 200, and the result I was getting listed close to the 100,000 accounts that I didn't want to be in the result. They had the right balances but I want a file that only has those balances for the accounts listed in file 2. This code below isn't really a part of my question, just showing something I've tried. I just think this is much easier and faster with a hash table now so I would like to go that route.
#Find and Rename the BALANCE and ACCOUNT NUMBER columns in both files.
$fileOne = import-csv "$scrubFileOne" -Delimiter "$scrubFileOneDelim" -Header (0..$useColumnsF1) | select -Property #{label="BALANCE";expression={$($_.$scrubFileOneBal)}},#{label="ACCT-NUM";expression={$($_.$scrubFileOneAcctNum)}}
$fileTwo = import-csv "$scrubFileTwo" -Delimiter "$scrubFileTwoDelim" -Header (0..$useColumnsF2) | select -Property #{label="BALANCE";expression={$($_.$scrubFileTwoBal)}},#{label="ACCT-NUM";expression={$($_.$scrubFileTwoAcctNum)}}
Compare-Object $fileOne $fileTwo -Property 'BALANCE','ACCTNUM' -IncludeEqual -PassThru | Where-Object{$_.sideIndicator -eq "<="} | select * -Exclude SideIndicator | export-csv -notype "C:\test\f1.txt"
What you are after is filtering the Compare-Object function. This will show only one side of the result. YOu will need to place this before you exclude that property for it to work.
| Where-Object{$_.sideIndicator -eq "<="} |
Assuming that you have the following hash tables:
$hash = #{
'000000000001' = '000000285+';
'000000000002' = '000031000+';
'000000000003' = '000004685+';
'000000000004' = '000025877+';
'000000000005' = '000000001+';
'000000000006' = '000031000+';
'000000000007' = '000018137+';
'000000000008' = '000000000+';
}
$hashTwo = #{
'000000000001' = '000008411+';
'000000000003' = '000018137+';
'000000000007' = '000042865+';
'000000000008' = '000009761+';
}
you can create the third hash table by iterating over the keys from the second hash table and then assigning those keys to the value from the first hash table.
$hashThree = #{}
ForEach ($key In $hashTwo.Keys) {
$hashThree["$key"] = $hash["$key"]
}
$hashThree
The output of $hashThree is:
Name Value
---- -----
000000000007 000018137+
000000000001 000000285+
000000000008 000000000+
000000000003 000004685+
If you want the order of the data maintained (and you are using PowerShell 6 Core), you can use [ordered]#{} when creating the hash tables.

re-arrange and combine powershell custom objects

I have a system that currently reads data from a CSV file produced by a separate system that is going to be replaced.
The imported CSV file looks like this
PS> Import-Csv .\SalesValues.csv
Sale Values AA BB
----------- -- --
10 6 5
5 3 4
3 1 9
To replace this process I hope to produce an object that looks identical to the CSV above, but I do not want to continue to use a CSV file.
I already have a script that reads data in from our database and extracts the data that I need to use. I'll not detail the fairly long script that preceeds this point but in effect it looks like this:
$SQLData = Custom-SQLFunction "SELECT * FROM SALES_DATA WHERE LIST_ID = $LISTID"
$SQLData will contain ~5000+ DataRow objects that I need to query.
One of those DataRow object looks something like this:
lead_id : 123456789
entry_date : 26/10/2018 16:51:16
modify_date : 01/11/2018 01:00:02
status : WRONG
user : mrexample
vendor_lead_code : TH1S15L0NGC0D3
source_id : A543212
list_id : 333004
list_name : AA Some Text
gmt_offset_now : 0.00
SaleValue : 10
list_name is going to be prefixed with AA or BB.
SaleValue can be any integer 3 and up, however realistically extremely unlikely to be higher than 100 (as this is a monthly donation) and will be one of 3,5,10 in the vast majority of occurrences.
I already have script that takes the content of list_name, creates and populates the data I need to use into two separate psobjects ($AASalesValues and $BBSalesValues) that collates the total numbers of 'SaleValue' across the data set.
Because I cannot reliably anticipate the value of any SaleValue I have to dynamically create the psobjects properties like this
foreach ($record in $SQLData) {
if ($record.list_name -match "BB") {
if ($record.SaleValue -gt 0) {
if ($BBSalesValues | Get-Member -Name $($record.SaleValue) -MemberType Properties) {
$BBSalesValues.$($record.SaleValue) = $BBSalesValues.$($record.SaleValue)+1
} else {
$BBSalesValues | Add-Member -Name $($record.SaleValue) -MemberType NoteProperty -Value 1
}
}
}
}
The two resultant objects look like this:
PS> $AASalesValues
10 5 3 50
-- - - --
17 14 3 1
PS> $BBSalesvalues
3 10 5 4
- -- - -
36 12 11 1
I now have the data that I need, however I need to format it in a way that replicates the format of the CSV so I can pass it directly to another existing powershell script that is configured to expect the data in the format that the CSV is in, but I do not want to write the data to a file.
I'd prefer to pass this directly to the next part of the script.
Ultimately what I want to do is to produce a new object/some output that looks like the output from Import-Csv command at the top of this post.
I'd like a new object, say $OverallSalesValues, to look like this:
PS>$overallSalesValues
Sale Values AA BB
50 1 0
10 17 12
5 14 11
4 0 1
3 3 36
In the above example the values from $AASalesValues is listed under the AA column, the values from $BBSalesValues is listed under the BB column, with the rows matching the headers of the two original objects.
I did try this with hashtables but I was unable to work out how to both create them from dynamic values and format them to how I needed them to look.
Finally got there.
$TotalList = #()
foreach($n in 3..200){
if($AASalesValues.$n -or $BBSalesValues.$n){
$AACount = $AASalesValues.$n
$BBcount = $BBSalesValues.$n
$values = [PSCustomObject]#{
'Sale Value'= $n
AA = $AACount
BB = $BBcount
}
$TotalList += $values
}
}
$TotalList
produces an output of
Sale Value AA BB
---------- -- --
3 3 36
4 2
5 14 11
10 18 12
50 1
Just need to add a bit to include '0' values instead of $null.
I'm going to assume that $record contains a list of the database results for either $AASalesValues or $BBSalesValues, not both, otherwise you'd need some kind of selector to avoid counting records of one group with the other group.
Group the records by their SaleValue property as LotPings suggested:
$BBSalesValues = $record | Group-Object SaleValue -NoElement
That will give you a list of the SaleValue values with their respective count.
PS> $BBSalesValues
Count Name
----- ----
36 3
12 10
11 5
1 4
You can then update your CSV data with these values like this:
$file = 'C:\path\to\data.csv'
# read CSV into a hashtable mapping the sale value to the complete record
# (so that we can lookup the record by sale value)
$csv = #{}
Import-Csv $file | ForEach-Object {
$csv[$_.'Sale Values'] = $_
}
# Add records for missing sale values
$($AASalesValues; $BBSalesValues) | Select-Object -Expand Name -Unique | ForEach-Object {
if (-not $csv.ContainsKey($_)) {
$csv[$_] = New-Object -Type PSObject -Property #{
'Sale Values' = $_
'AA' = 0
'BB' = 0
}
}
}
# update records with values from $AASalesValues
$AASalesValues | ForEach-Object {
[int]$csv[$_.Name].AA += $_.Count
}
# update records with values from $BBSalesValues
$BBSalesValues | ForEach-Object {
[int]$csv[$_.Name].BB += $_.Count
}
# write updated records back to file
$csv.Values | Export-Csv $file -NoType
Even with your updated question the approach would be pretty much the same, you'd just add another level of grouping for collecting the sales numbers:
$sales = #{}
$record | Group-Object {$_.list_name.Split()[0]} | ForEach-Object {
$sales[$_.Name] = $_.Group | Group-Object SaleValue -NoElement
}
and then adjust the merging to something like this:
$file = 'C:\path\to\data.csv'
# read CSV into a hashtable mapping the sale value to the complete record
# (so that we can lookup the record by sale value)
$csv = #{}
Import-Csv $file | ForEach-Object {
$csv[$_.'Sale Values'] = $_
}
# Add records for missing sale values
$sales.Values | Select-Object -Expand Name -Unique | ForEach-Object {
if (-not $csv.ContainsKey($_)) {
$prop = #{'Sale Values' = $_}
$sales.Keys | ForEach-Object {
$prop[$_] = 0
}
$csv[$_] = New-Object -Type PSObject -Property $prop
}
}
# update records with values from $sales
$sales.GetEnumerator() | ForEach-Object {
$name = $_.Key
$_.Value | ForEach-Object {
[int]$csv[$_.Name].$name += $_.Count
}
}
# write updated records back to file
$csv.Values | Export-Csv $file -NoType

powershell compare two files and list their columns with side indicator as match/mismatch

I have seen powershell script which also I have in mind. What I would like to add though is another column which would show the side indicator comparators ("==", "<=", "=>") and be named them as MATCH(if "==") and MISMATCH(if "<=" and "=>").
Any advise on how I would do this?
Here is the link of the script (Credits to Florent Courtay)
How can i reorganise powershell's compare-object output?
$a = Compare-Object (Import-Csv 'C:\temp\f1.csv') (Import-Csv 'C:\temp\f2.csv') -property Header,Value
$a | Group-Object -Property Header | % { New-Object -TypeName psobject -Property #{Header=$_.name;newValue=$_.group[0].Value;oldValue=$_.group[1].Value}}
========================================================================
The output I have in mind:
Header1 Old Value New Value STATUS
------ --------- --------- -----------
String1 Value 1 Value 2 MATCH
String2 Value 3 Value 4 MATCH
String3 NA Value 5 MISMATCH
String4 Value 6 NA MISMATCH
Here's a self-contained solution; simply replace the ConvertFrom-Csv calls with your Import-Csv calls:
# Sample CSV input.
$csv1 = #'
Header,Value
a,1
b,2
c,3
'#
$csv2 = #'
Header,Value
a,1a
b,2
d,4
'#
Compare-Object (ConvertFrom-Csv $csv1) (ConvertFrom-Csv $csv2) -Property Header, Value |
Group-Object Header | Sort-Object Name | ForEach-Object {
$newValIndex, $oldValIndex = ((1, 0), (0, 1))[$_.Group[0].SideIndicator -eq '=>']
[pscustomobject] #{
Header = $_.Name
OldValue = ('NA', $_.Group[$oldValIndex].Value)[$null -ne $_.Group[$oldValIndex].Value]
NewValue = ('NA', $_.Group[$newValIndex].Value)[$null -ne $_.Group[$newValIndex].Value]
Status = ('MISMATCH', 'MATCH')[$_.Group.Count -gt 1]
}
}
The above yields:
Header OldValue NewValue Status
------ -------- -------- ------
a 1 1a MATCH
c 3 NA MISMATCH
d NA 4 MISMATCH
Note:
The assumption is that a given Header column value appears at most once in each input file.
The Sort-Object Name call is needed to sort the output by Header valuesThanks, LotPings.
, because, due to how Compare-Object orders its output (right-side-only items first), the order of groups created by Group-Object would not automatically reflect the 1st CSV's order of header values (d would appear before c).

Pivot an array of array?

The following code defined a matrix.
$a = #('a','b','x',10),
#('a','b','y',20),
#('c','e','x',50),
#('c','e','y',30)
$a | % { "[$_]"}
I want to pivot the array by x and y. The expected result array should be
[a b 10 20]
[c e 50 30]
- - -- --
x y
I think it needs group-object and then mapping. How to use group-object on an array?
(BTW, why the question has been down voted twice?)
You can't use Group-Object with an array (at least not the way you want) since Group-Object works on object properties. A workaround is to organize your rows into a label that you want to group on, followed by the values to assign to the group. Then you can group on the label:
$a | %{
new-object PsObject -prop #{"label" = "$($_[0]),$($_[1])"; value=#{ $_[2]=$_[3]}}
} | Group-Object label
So, then you have a group with your entries stroed as an array of hashtables within each group:
Count Name Group
----- ---- -----
2 a,b {#{value=System.Collections.Hashtable; label=a,b}, #{value=System.Collections.Hashtable; label=a,b}}
2 c,e {#{value=System.Collections.Hashtable; label=c,e}, #{value=System.Collections.Hashtable; label=c,e}}
You can then expand out each row to get the info you desire:
$a | %{
new-object PsObject -prop #{"label" = "$($_[0]),$($_[1])"; value=#{ $_[2]=$_[3]}}
} |
group label | % {
"[$(#($_.Name -split ",") + #($_.Group.value.values))]"
}
which gives:
[a b 10 20]
[c e 50 30]
To answer your second comment, no the above won;t guarantee the order. To guarantee it, you'll have to be explicit:
$a | %{
new-object PsObject -prop #{"label" = "$($_[0]),$($_[1])"; value=#{ $_[2]=$_[3]}}
} |
group label | % {
"[$(#($_.Name -split ",") + #($_.Group.value.x, $_.Group.value.y))]"
}