Getting an average of each column from CSV - powershell

Question

I am trying to get the average of all the columns in CSV with respect to timestamp. The object type is system.array. Whenever I try to convert the integer it is showing an error.
timestamp streams TRP A B C D
6/4/2019 6775 305 56 229 132 764
6/4/2019 6910 316 28 356 118 134
6/4/2019 6749 316 54 218 206 144
6/5/2019 5186 267 84 280 452 258
6/5/2019 5187 240 33 436 455 245
6/5/2019 5224 291 21 245 192 654
6/6/2019 5254 343 42 636 403 789
6/6/2019 5180 252 23 169 328 888
6/6/2019 5181 290 32 788 129 745
6/6/2019 5244 328 44 540 403 989
I got help from Lee_Dailey on the below code, I was trying to produce the average of each column based on timestamp. I get an error
Cannot convert value " " to type "System.Int32".
Error: "Index was outside the bounds of the array."
+ ... l = [Math]::Round(($GIS_Item.Group.$TPL_Item.ForEach({[int]$_}) | Mea ...
+ ~~~~~~~
+ CategoryInfo : InvalidArgument: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvalidCastFromStringToInteger
$InStuff = Import-Csv 'M:\MyDoc\script\logfiles\Output_18Mar\streams_E1WAF2_OUTPUT.csv'
$TargetPropertyList = $InStuff[0].PSObject.Properties.Name.Where({$_ - ne 'TimeStamp'})
$GroupedInStuff = $InStuff | Group-Object -Property TimeStamp
$Results = foreach ($GIS_Item in $GroupedInStuff) {
$HighestValues = [ordered]#{
TimeStamp = $GIS_Item.Name
}
foreach ($TPL_Item in $TargetPropertyList) {
$TempHiVal = [Math]::Round(($GIS_Item.Group.$TPL_Item.ForEach({[int]$_}) | Measure-Object -Average).Average)
$HighestValues.Add($TPL_Item, $TempHiVal)
}
[PSCustomObject]$HighestValues
}
$Results = $Results | Sort-Object -Property {[DateTime]$_.TimeStamp}

Answer 1

here's a way to deal with the somewhat damaged CSV file you presented in the new data set. [grin] it reads the file in as plain text, removes the spaces, and trims the final |.
i was wondering if having only one date in a grouping was a problem, so i added a final line to the data set that has a different date.
# fake reading in a defective CSV file as plain text
# in real life, use Get-Content
$InStuff = #'
timestamp|abc | A | B | C | D | E | F | G |
6/4/2019 |6775 | 3059 | 4 | 2292 | 1328 | 764 | 0 | 0 |
6/4/2019 |6910 | 3167 | 28 | 3568 | 1180 | 1348 | 0 | 0 |
6/4/2019 |6749 | 3161 | 0 | 2180 | 2060 | 1440 | 0 | 28 |
6/5/2019 |6738 | 3118 | 4 | 2736 | 1396 | 984 | 0 | 0 |
6/5/2019 |6718 | 3130 | 12 | 3076 | 1008 | 452 | 0 | 4 |
6/5/2019 |6894 | 3046 | 4 | 2284 | 1556 | 624 | 0 | 0 |
1/1/2021 |1111 | 2222 | 3 | 4444 | 5555 | 666 | 7 | 8 |
'# -split [System.Environment]::NewLine
$CleanedInStuff = $InStuff.ForEach({$_.Replace(' ', '').Trim('|')}) |
ConvertFrom-Csv -Delimiter '|'
$TargetPropertyList = $CleanedInStuff[0].PSObject.Properties.Name.
Where({
$_ -ne 'TimeStamp'
})
$GroupedCIS = $CleanedInStuff |
Group-Object -Property TimeStamp
$Results = foreach ($GCIS_Item in $GroupedCIS) {
$TempObject = [ordered]#{
TimeStamp = $GCIS_Item.Name
}
foreach ($TPL_Item in $TargetPropertyList) {
$TempAveValue = [Math]::Round(($GCIS_Item.Group.$TPL_Item.
ForEach({[int]$_}) |
Measure-Object -Average).Average, 2)
$TempObject.Add($TPL_Item, $TempAveValue)
}
[PSCustomObject]$TempObject
}
$Results = $Results |
Sort-Object -Property {
[DateTime]$_.TimeStamp
}
$Results
output ...
TimeStamp : 6/4/2019
abc : 6811.33
A : 3129
B : 10.67
C : 2680
D : 1522.67
E : 1184
F : 0
G : 9.33
TimeStamp : 6/5/2019
abc : 6783.33
A : 3098
B : 6.67
C : 2698.67
D : 1320
E : 686.67
F : 0
G : 1.33
TimeStamp : 1/1/2021
abc : 1111
A : 2222
B : 3
C : 4444
D : 5555
E : 666
F : 7
G : 8

Answer 2

There is probably a nicer way to do this that Lee will surely advise, but this is how I would complete the task. This will group the objects by the Timestamp property and then you can average them there. I recommend playing around with $csv | Group-Object timestamp and see what you can do with it.
$csv = import-csv C:\temp\test.csv
$Averages = New-Object System.Collections.ArrayList
Foreach($object in ($csv | Group-Object timestamp)) {
$Averages.Add([pscustomobject]#{
timestamp = $object.Name
abc = ($object.group | Select-Object -ExpandProperty abc | Measure-Object -Average).Average
a = ($object.group | Select-Object -ExpandProperty a | Measure-Object -Average).Average
b = ($object.group | Select-Object -ExpandProperty b | Measure-Object -Average).Average
c = ($object.group | Select-Object -ExpandProperty c | Measure-Object -Average).Average
d = ($object.group | Select-Object -ExpandProperty d | Measure-Object -Average).Average
e = ($object.group | Select-Object -ExpandProperty e | Measure-Object -Average).Average
f = ($object.group | Select-Object -ExpandProperty f | Measure-Object -Average).Average
g = ($object.group | Select-Object -ExpandProperty g | Measure-Object -Average).Average
})
}
Output:
PS H:\> $Averages
timestamp : 6/4/2019
abc : 6811.33333333333
a : 3129
b : 10.6666666666667
c : 2680
d : 1522.66666666667
e : 1184
f : 0
g : 9.33333333333333
timestamp : 6/5/2019
abc : 6783.33333333333
a : 3098
b : 6.66666666666667
c : 2698.66666666667
d : 1320
e : 686.666666666667
f : 0
g : 1.33333333333333

Answer 3

The ConvertFrom-SourceTable cmdlet I created is actually supposed to read fixed width column tables but there isn't any reason why it shouldn't be able to also read delimited (or distorted) tables. So this question has encouraged me to create an update which no longer produces errors when this happens.
Converting the table in the question
$Table = '
timestamp streams TRP A B C D
6/4/2019 6775 305 56 229 132 764
6/4/2019 6910 316 28 356 118 134
6/4/2019 6749 316 54 218 206 144
6/5/2019 5186 267 84 280 452 258
6/5/2019 5187 240 33 436 455 245
6/5/2019 5224 291 21 245 192 654
6/6/2019 5254 343 42 636 403 789
6/6/2019 5180 252 23 169 328 888
6/6/2019 5181 290 32 788 129 745
6/6/2019 5244 328 44 540 403 989
'
# Raw Table
ConvertFrom-SourceTable $Table | Format-Table
timestamp streams TRP A B C D
--------- ------- --- - - - -
6/4/2019 6775 305 56 229 132 764
6/4/2019 6910 316 28 356 118 134
6/4/2019 6749 316 54 218 206 144
6/5/2019 5186 267 84 280 452 258
6/5/2019 5187 240 33 436 455 245
6/5/2019 5224 291 21 245 192 654
6/6/2019 5254 343 42 636 403 789
6/6/2019 5180 252 23 169 328 888
6/6/2019 5181 290 32 788 129 745
6/6/2019 5244 328 44 540 403 989
#Streamed rows from pipeline:
$Table -split [System.Environment]::NewLine | ConvertFrom-SourceTable | Format-Table
timestamp streams TRP A B C D
--------- ------- --- - - - -
6/4/2019 6775 305 56 229 132 764
6/4/2019 6910 316 28 356 118 134
6/4/2019 6749 316 54 218 206 144
6/5/2019 5186 267 84 280 452 258
6/5/2019 5187 240 33 436 455 245
6/5/2019 5224 291 21 245 192 654
6/6/2019 5254 343 42 636 403 789
6/6/2019 5180 252 23 169 328 888
6/6/2019 5181 290 32 788 129 745
6/6/2019 5244 328 44 540 403 989
Fixed width column table with vertical rulers
$Table = '
| date | abc | A | B | C | D | E | F | G |
| 6/4/2019 | 6775 | 3059 | 4 | 2292 | 1328 | 764 | 0 | 0 |
| 6/4/2019 | 6910 | 3167 | 28 | 3568 | 1180 | 1348 | 0 | 0 |
| 6/4/2019 | 6749 | 3161 | 0 | 2180 | 2060 | 1440 | 0 | 28 |
| 6/5/2019 | 6738 | 3118 | 4 | 2736 | 1396 | 984 | 0 | 0 |
| 6/5/2019 | 6718 | 3130 | 12 | 3076 | 1008 | 452 | 0 | 4 |
| 6/5/2019 | 6894 | 3046 | 4 | 2284 | 1556 | 624 | 0 | 0 |
| 1/1/2021 | 1111 | 2222 | 3 | 4444 | 5555 | 666 | 7 | 8 |
'
# Raw Table
ConvertFrom-SourceTable $Table | Format-Table
date abc A B C D E F G
---- --- - - - - - - -
6/4/2019 6775 3059 4 2292 1328 764 0 0
6/4/2019 6910 3167 28 3568 1180 1348 0 0
6/4/2019 6749 3161 0 2180 2060 1440 0 28
6/5/2019 6738 3118 4 2736 1396 984 0 0
6/5/2019 6718 3130 12 3076 1008 452 0 4
6/5/2019 6894 3046 4 2284 1556 624 0 0
1/1/2021 1111 2222 3 4444 5555 666 7 8
#Streamed rows from pipeline:
$Table -split [System.Environment]::NewLine | ConvertFrom-SourceTable | Format-Table
date abc A B C D E F G
---- --- - - - - - - -
6/4/2019 6775 3059 4 2292 1328 764 0 0
6/4/2019 6910 3167 28 3568 1180 1348 0 0
6/4/2019 6749 3161 0 2180 2060 1440 0 28
6/5/2019 6738 3118 4 2736 1396 984 0 0
6/5/2019 6718 3130 12 3076 1008 452 0 4
6/5/2019 6894 3046 4 2284 1556 624 0 0
1/1/2021 1111 2222 3 4444 5555 666 7 8
Note the type casting (table alignment) in the results, meaning that the result is symmetrical:
$Result = $Table | ConvertFrom-SourceTable | Format-Table
$Result | Format-Table <=> $Result | ConvertFrom-SourceTable | Format-Table
Distorted table
$Table = '
timestamp|abc | A | B | C | D | E | F | G |
6/4/2019 |6775 | 3059 | 4 | 2292 | 1328 | 764 | 0 | 0 |
6/4/2019 |6910 | 3167 | 28 | 3568 | 1180 | 1348 | 0 | 0 |
6/4/2019 |6749 | 3161 | 0 | 2180 | 2060 | 1440 | 0 | 28 |
6/5/2019 |6738 | 3118 | 4 | 2736 | 1396 | 984 | 0 | 0 |
6/5/2019 |6718 | 3130 | 12 | 3076 | 1008 | 452 | 0 | 4 |
6/5/2019 |6894 | 3046 | 4 | 2284 | 1556 | 624 | 0 | 0 |
1/1/2021 |1111 | 2222 | 3 | 4444 | 5555 | 666 | 7 | 8 |
'
# Raw Table
ConvertFrom-SourceTable $Table | Format-Table
timestamp abc A B C D E F G
--------- --- - - - - - - -
6/4/2019 6775 3059 4 2292 1328 764 0 0
6/4/2019 6910 3167 28 3568 1180 1348 0 0
6/4/2019 6749 3161 0 2180 2060 1440 0 28
6/5/2019 6738 3118 4 2736 1396 984 0 0
6/5/2019 6718 3130 12 3076 1008 452 0 4
6/5/2019 6894 3046 4 2284 1556 624 0 0
1/1/2021 1111 2222 3 4444 5555 666 7 8
#Streamed rows from pipeline:
$Table -split [System.Environment]::NewLine | ConvertFrom-SourceTable | Format-Table
timestamp abc A B C D E F G
--------- --- - - - - - - -
6/4/2019 6775 3059 4 2292 1328 764 0 0
6/4/2019 6910 3167 28 3568 1180 1348 0 0
6/4/2019 6749 3161 0 2180 2060 1440 0 28
6/5/2019 6738 3118 4 2736 1396 984 0 0
6/5/2019 6718 3130 12 3076 1008 452 0 4
6/5/2019 6894 3046 4 2284 1556 624 0 0
1/1/2021 1111 2222 3 4444 5555 666 7 8
Note that distorted rows will always result in a literal (string) conversion (no type casting)