Working with CSV files - powershell

Suppose I have a table like this in data.csv:
time channel x y z
0.001 point 1 1 2 3
0.001 point 2 4 5 6
0.001 point 3 7 8 9
0.001 point 4 10 11 12
0.001 point 5 13 14 15
0.002 point 1 2 3 4
0.002 point 2 5 6 7
0.002 point 3 8 9 10
0.002 point 4 11 12 13
0.002 point 5 14 15 16
0.004 point 1 3 4 5
0.004 point 2 6 7 8
0.004 point 3 9 10 11
0.004 point 4 12 13 14
0.004 point 5 15 16 17
How do I make Powershell to write out 3 files (Xdata.csv, Ydata.csv, Zdata.csv), which Xdata.csv should look like this:
time point 1 point 2 point 3 point 4 point 5
0.001 1 4 7 10 13
0.002 2 5 8 11 14
0.004 3 6 9 12 15
So far, my code looks like this:
# Path to current directory
$path = [System.IO.Path]::GetDirectoryName($myInvocation.MyCommand.Definition)
# csv filename to be imported from
$filename = "data.csv"
$data = Import-Csv "$path\$filename"
# Array of unique values of times
$times = $data | % { $_.time } | Sort-Object | Get-Unique
# Array of unique values of channels
$channels = $data | % { $_.channel } | Sort-Object | Get-Unique
But at this point, I am struggling of how to set up an output table just like the one above.

I would use Group-Object + some logic to generate objects using data from each 'time' snapshot
$collections = #{
x = #()
y = #()
z = #()
}
$data | Group time | ForEach-Object {
$x = New-Object PSObject -Property #{
time = $_.Name
}
$y = New-Object PSObject -Property #{
time = $_.Name
}
$z = New-Object PSObject -Property #{
time = $_.Name
}
foreach ($item in $_.Group) {
if ($item.channel) {
$x | Add-Member -MemberType NoteProperty -Name $item.channel -Value $item.x
$y | Add-Member -MemberType NoteProperty -Name $item.channel -Value $item.y
$z | Add-Member -MemberType NoteProperty -Name $item.channel -Value $item.z
}
}
$collections.x += $x
$collections.y += $y
$collections.z += $z
}
foreach ($coordinate in 'x','y','z') {
$collections.$coordinate | Export-Csv -NoTypeInformation "${coordinate}data.csv"
}
This is with assumption that 'data' contains object similar to one I could generate with:
New-Object PSObject -Property #{
time = 0.001
channel = 'point 1'
x = 1
y = 2
z = 3
}

More of a pipeline approach (not tested)
$axes = #{
x = [ordered]#{}
y = [ordered]#{}
z = [ordered]#{}
}
import-csv data.csv |
foreach-object {
if ( $axes.x.containskey($_.time) )
{
foreach ($axis in 'x','y','z')
{ $axes.$axis[$_.time].add($_.channel,$_.$axis) }
}
else {
foreach ($axis in 'x','y','z')
{ $axes.$axis[$_.time] = [ordered]#{ time = $_.time;$_.channel = $_.$axis } }
}
}
foreach ($axis in 'x','y','z')
{
$(foreach ($time in $axes.$axis.values)
{ [PSCustomObject]$time })|
export-csv ${$axis}data.csv -NoTypeInformation
}

Related

Get random items from hashtable but the total of values has to be equal to a set number

I'm trying to build a simple "task distributor" for the house tasks between me and my wife. Although the concept will be really useful at work too so I need to learn it properly.
My hashtable:
$Taches = #{
"Balayeuse plancher" = 20
"Moppe plancher" = 20
"Douche" = 15
"Litières" = 5
"Poele" = 5
"Comptoir" = 5
"Lave-Vaisselle" = 10
"Toilette" = 5
"Lavabos" = 10
"Couvertures lit" = 5
"Poubelles" = 5
}
The total value for all the items is 105 (minutes).
So roughly 50mins each of we split it in two.
My goal:
I want to select random items from that hashtable and build two different hashtables - one for me and my wife, each having a total value of 50 (So it's fair). For example 20+20+10 or 5+5+5+15+20, etc. The hard part is that ALL tasks have to be accounted for between the two hashtables and they can only be present ONCE in each of them (no use in cleaning the same thing twice!).
What would be the best option?
For now I successfully achieved a random hashtable of a total value of 50 like this:
do {
$Me = $null
$sum = $null
$Me = #{}
$Me = $Taches.GetEnumerator() | Get-Random -Count 5
$Me | ForEach-Object { $Sum += $_.value }
} until ($sum -eq 50)
Result example :
Name Value
---- -----
Poubelles 5
Balayeuse plancher 20
Douche 15
Poele 5
Toilette 5
It works but boy does it feel like it's a roundabout and crooked way of doing it. I'm sure there is a better approach? Plus I'm lacking important things. ALL the tasks have to be accounted for and not be present twice. This is quite complicated although it looked simple at first!
You can not maximise randomness and fairness at the same time so one has to give. I think you should not risk being unfair to your wife and so fairness must prevail!
Fairness at the expense of randomness
This approach sorts the items in descending time order and then randomly assigns them items to each person unless that assignment would be unfair.
The fairness calculation here is that the maximum time difference should be at most the duration of the quickest task.
$DescendingOrder = $Taches.Keys | Sort-Object -Descending { $Taches[$_] }
$Measures = $Taches.Values | Measure-Object -Sum -Minimum
$UnfairLimit = ($Measures.Sum + $Measures.Minimum) / 2
$Person1 = #{}
$Person2 = #{}
$Total1 = 0
$Total2 = 0
foreach ($Item in $DescendingOrder) {
$Time = $Taches[$Item]
$Choice = Get-Random 2
if (($Choice -eq 0) -and (($Total1 + $Time) -gt $UnfairLimit)) {
$Choice = 1
}
if (($Choice -eq 1) -and (($Total2 + $Time) -gt $UnfairLimit)) {
$Choice = 0
}
if ($Choice -eq 0) {
$Person1[$Item] = $Time
$Total1 += $Time
} else {
$Person2[$Item] = $Time
$Total2 += $Time
}
}
An example run:
PS> $Person1 | ConvertTo-Json
{
"Comptoir": 5,
"Lavabos": 10,
"Litières": 5,
"Couvertures lit": 5,
"Douche": 15,
"Lave-Vaisselle": 10
}
and the other person:
PS> $Person2 | ConvertTo-Json
{
"Moppe plancher": 20,
"Toilette": 5,
"Balayeuse plancher": 20,
"Poubelles": 5,
"Poele": 5
}
Randomness at the expense of fairness
This approach is to randomize the list, go through each item and then assign it to the person who has the least time allocated to them so far.
Earlier decisions might mean that later decisions end up being unfair.
$RandomOrder = $Taches.Keys | Sort-Object { Get-Random }
$Person1 = #{}
$Person2 = #{}
$Total1 = 0
$Total2 = 0
foreach ($Item in $RandomOrder) {
$Time = $Taches[$Item]
if ($Total1 -lt $Total2) {
$Person1[$Item] = $Time
$Total1 += $Time
} else {
$Person2[$Item] = $Time
$Total2 += $Time
}
}
An example run:
PS> $Person1 | ConvertTo-Json
{
"Poele": 5,
"Douche": 15,
"Couvertures lit": 5,
"Lave-Vaisselle": 10,
"Balayeuse plancher": 20,
"Toilette": 5
}
and the other person:
PS> $Person2 | ConvertTo-Json
{
"Lavabos": 10,
"Comptoir": 5,
"Poubelles": 5,
"Litières": 5,
"Moppe plancher": 20
}
You should probably write the algorithm to always have you take the extra task in a rounding error (Happy Wife, Happy Life).
This is probably over-engineered, but I was intrigued by the question, and learned some French in the process.
$Taches = #{
"Balayeuse plancher" = 20
"Moppe plancher" = 20
"Douche" = 15
"Litières" = 5
"Poele" = 5
"Comptoir" = 5
"Lave-Vaisselle" = 10
"Toilette" = 5
"Lavabos" = 10
"Couvertures lit" = 5
"Poubelles" = 5
}
$target = 0
$epsilon = 5
# copy if you don't want to destroy original list (not needed probably)
# put all entries in first list.
# randomly move entry to p2 if count over target +/- epsilon
# randomly move entry from p2 if count under target +/- epsilon
# (unless you know you can always get exactly target and not loop forever trying)
$p1 = #{} # person 1
$p2 = #{} # person 2
$p1Total = 0 # optimizaton to not have to walk entire list and recalculate constantly
$p2Total = 0 # might as well track this too...
$Taches.Keys | % {
$p1.Add($_, $Taches[$_])
$p1Total += $Taches[$_]
$target += $Taches[$_]
}
$target = $target / 2
$done = $false
while (-not $done)
{
if ($p1Total -gt ($target+$epsilon))
{
$item = $p1.Keys | Get-Random
$value = $p1[$item]
$p1.Remove($item)
$p2.Add($item, $value)
$p1Total -= $value
$p2Total += $value
continue
}
elseif ($p1Total -lt ($target-$epsilon))
{
$item = $p2.Keys | Get-Random
$value = $p2[$item]
$p2.Remove($item)
$p1.Add($item, $value)
$p1Total += $value
$p2Total -= $value
continue
}
$done = $true
}
"Final result"
"p1"
$p1Total
$p1
"`np2"
$p2Total
$p2
Yet another approach:
$MinSum = ($Taches.Values | Measure-Object -Minimum ).Minimum
$HalfSum = ($Taches.Values | Measure-Object -Sum ).Sum / 2
do {
$sum = 0
$All = $Taches.GetEnumerator() |
Get-Random -Count $Taches.Keys.Count
$Me = $All | ForEach-Object {
if ( $Sum -lt $HalfSum - $MinSum ) {
$Sum += $_.value
#{ $_.Key = $_.Value }
}
}
Write-Host "$sum " -NoNewline # debugging output
} until ($sum -eq 50 )
$Em = $Taches.Keys | ForEach-Object {
if ( $_ -notin $Me.Keys ) {
#{ $_ = $Taches.$_ }
}
}
# show "fairness" (task count vs. task cost)
$Me.Values | Measure-Object -Sum | Select-Object -Property Count, Sum
$Em.Values | Measure-Object -Sum | Select-Object -Property Count, Sum
Sample output(s):
PS D:\PShell> D:\PShell\SO\54610011.ps1
50
Count Sum
----- ---
4 50
7 55
PS D:\PShell> D:\PShell\SO\54610011.ps1
65 65 50
Count Sum
----- ---
6 50
5 55
Great answers guys, learned a lot. Here is what I ended up doing thanks to "Fischfreund" on Reddit (https://www.reddit.com/r/PowerShell/comments/aovs8s/get_random_items_from_hashtable_but_the_total_of/eg3ytds).
His approach is amazingly simple yet I didn't think of it at all.
First hashtable : Get a random count of 5 until the sum is 50. Then create a second hashtable where the items are not in the first hashtable! I assign that first hahstable containing 5 items to my wife so I'm the one who always has an extra task (like suggested by Kory ;)). Phew i'm safe.
$Taches = #{
"Balayeuse plancher" = 20
"Moppe plancher" = 20
"Douche" = 15
"Litières" = 5
"Poele" = 5
"Comptoir" = 5
"Lave-Vaisselle" = 10
"Toilette" = 5
"Lavabos" = 10
"Couvertures lit" = 5
"Poubelles" = 5
}
do {
$Selection1 = $Taches.GetEnumerator() | Get-Random -Count 5
} until (($Selection1.Value | measure -Sum ).Sum -eq 50)
$Selection2 = $Taches.GetEnumerator() | Where-Object {$_ -notin $Selection1}
$Selection1 | select-object #{Name="Personne";expression={"Wife"} },Name,Value
""
$Selection2 | select-object #{Name="Personne";expression={"Me"} },Name,Value

randomly change a 2D array based on percentage in Powershell

I'm trying to randomly change an array based on a variable percentage in Powershell.
unfortunaly I dont get the expected output.
In the script below I'm trying to change 20% of the array based on a random generated output.
Does somebody know the best way to acomplisch this?
$row = 4
$col = 2
$temp_array = New-Object 'object[,]' $row,$col
for ($i=0;$i -le $row -1 ; $i++) {
for ($j=0;$j -le $col -1 ; $j++) {
$temp_array[$i,$j] = Get-Random -Minimum 1 -Maximum 10
}
}
$perc_to_change = 0.2
for ($i=0;$i -le $row -1 ; $i++) {
for ($j=0;$j -le $col -1 ; $j++) {
$check_digit = Get-Random -Maximum 1 -Minimum 0.0
If($check_digit -lt $perc_to_change){
$temp_array[$i,$j] = "changed"
}
Write-Host "temp array [" $i "][" $j "] = " $temp_array[$i,$j]
}
}
IMO you need a different apprach to get the random percentage spread over rows and columns.
Uniquely number rows and columns
C0 C1 C2 C3 C4
R0 0 1 2 3 4
R1 5 6 7 8 9
R2 10 11 12 13 14
R3 15 16 17 18 19
R4 20 21 22 23 24
This script uses Get-Random -Count to select the percentage from the multiplication of $rows by $cols, iterates the results and recalculates row and column.
## Q:\Test\2018\10\23\SO_52949282.ps1
$row = 5
$col = 5
$temp_array = New-Object 'object[,]' $row,$col
for ($i=0;$i -le $row -1 ; $i++) {
for ($j=0;$j -le $col -1 ; $j++) {
$temp_array[$i,$j] = Get-Random -Minimum 1 -Maximum 10
}
}
$elements = $row * $col
$Count = [Math]::Floor($elements * 0.2)
0..$($elements-1) | Get-Random -Count $count | ForEach-Object {
"{0,2} = [{1},{2}]" -f $_,[Math]::Floor($_/$col),($_ % $col )
}
What you do with the resulting percentage I'll leave up to you.
Sample output (20% of 5*5 is 5):
6 = [1,1]
11 = [2,1]
16 = [3,1]
9 = [1,4]
19 = [3,4]
for what i meant by a better match to your exactly 20% request, take a look at this ...
$TargetPct = 20
$ValueRange = #(20, 40, 60, 80, 100)
$Results = 1..100 |
ForEach-Object {
Get-Random -InputObject $ValueRange
}
($Results -eq $TargetPct).Count
the above will show counts that are consistently near the ideal 20% of 100. i ran the code several times and the count of "exactly 20" ranged from 17 to 23.
using the -InputObject parameter gets around the "off by one" that the -Max parameter gives you.
-Min 10 -Max 20 >>> 10..19
-InputObject (10..20) >>> 10..20

Conditional criteria in powershell group measure-object?

I have data in this shape:
externalName,day,workingHours,hoursAndMinutes
PRJF,1,11,11:00
PRJF,2,11,11:00
PRJF,3,0,0:00
PRJF,4,0,0:00
CFAW,1,11,11:00
CFAW,2,11,11:00
CFAW,3,11,11:00
CFAW,4,11,11:00
CFAW,5,0,0:00
CFAW,6,0,0:00
and so far code is
$gdata = Import-csv $filepath\$filename | Group-Object -Property Externalname;
$test = #()
$test += foreach($rostername in $gdata) {
$rostername.Group | Select -Unique externalName,
#{Name = 'AllDays';Expression = {(($rostername.Group) | measure -Property day).count}},
}
$test;
What I can't work out is how to do a conditional count of the lines where day is non-zero.
The aim is to produce two lines:
PRJF, 4, 2, 11
CFAW, 6, 4, 11
i.e. Roster name, roster length, days on, average hours worked per day on.
You need a where-object to filter for non zero workinghours
I'd use a [PSCustomObject] to generate a new table
EDIT a bit more efficient with only one Measure-Object
## Q:\Test\2018\08\06\SO_51700660.ps1
$filepath = 'Q:\Test\2018\08\06'
$filename = 'SO_S1700660.csv'
$gdata = Import-Csv (Join-Path $filepath $filename) | Group-Object -Property Externalname
$test = ForEach($Roster in $gdata) {
$WH = ($Roster.Group.Workinghours|Where-Object {$_ -ne 0}|Measure-Object -Ave -Sum)
[PSCustomObject]#{
RosterName = $Roster.Name
RosterLength = $Roster.Count
DaysOn = $WH.count
AvgHours = $WH.Average
TotalHours = $WH.Sum
}
}
$test | Format-Table
Sample output:
> .\SO_51700660.ps1
RosterName RosterLength DaysOn AvgHours TotalHours
---------- ------------ ------ -------- ----------
PRJF 4 2 11 22
CFAW 6 4 11 44

Powershell: Combine single arrays into columns

Given:
$column1 = #(1,2,3)
$column2 = #(4,5,6)
How can I combine them into an object $matrix which gets displayed as a matrix with the single arrays as columns:
column1 column2
------- -------
1 4
2 5
3 6
It seems that all of my solutions today requires calculated properties. Try:
$column1 = #(1,2,3)
$column2 = #(4,5,6)
0..($column1.Length-1) | Select-Object #{n="Id";e={$_}}, #{n="Column1";e={$column1[$_]}}, #{n="Column2";e={$column2[$_]}}
Id Column1 Column2
-- ------- -------
0 1 4
1 2 5
2 3 6
If the lengths of the arrays are not equal, you could use:
$column1 = #(1,2,3)
$column2 = #(4,5,6,1)
$max = ($column1, $column2 | Measure-Object -Maximum -Property Count).Maximum
0..$max | Select-Object #{n="Column1";e={$column1[$_]}}, #{n="Column2";e={$column2[$_]}}
I wasn't sure if you needed the Id, so I included it in the first sample to show how to include it.
Little better, maybe:
$column1 = #(1,2,3)
$column2 = #(4,5,6,7)
$i=0
($column1,$column2 | sort length)[1] |
foreach {
new-object psobject -property #{
loess = $Column1[$i]
lowess = $column2[$i++]
}
} | ft -auto
loess lowess
----- ------
1 4
2 5
3 6
7
Here's something I created today. It takes a range of 0 to one of the column lengths, then maps it to a list of hashes. Use the select to turn it into a proper table.
$table = 0..$ColA.Length | % { #{
ColA = $ColA[$_]
ColB = $ColB[$_]
}} | Select ColA, ColB
Using the following variables:
$ColA = #(1, 2, 3)
$ColB = #(4, 5, 6)
Results in
ColB ColA
---- ----
1 4
2 5
3 6
I came up with this.. but it seems too verbose. Anything shorter?
&{
for ($i=0; $i -lt $y.Length; $i++) {
New-Object PSObject -Property #{
y = $y[$i]
loess = $smooth_loess[$i]
lowess = $smooth_lowess[$i]
}
}
} | Format-Table -AutoSize
Here is a combination of mjolinor and Frode F. solutions. I ran into some problems using Frode's object construction trick using select-object. For some reason it would output hash values likely representing object references. I only code in PowerShell a few times a year, so I am just providing this in case anyone else finds it useful (perhaps even my future self).
$column1 = #(1,2,3)
$column2 = #(4,5,6,7)
$column3 = #(2,5,5,2,1,3);
$max = (
$column1,
$column2,
$column3 |
Measure-Object -Maximum -Property Count).Maximum;
$i=0
0..$max |
foreach {
new-object psobject -property #{
col1 = $Column1[$i]
col3 = $column3[$i]
col2 = $column2[$i++]
}
} | ft -auto

String split with PowerShell

I have a log file what contains S.M.A.R.T. data of my hard drive.
I would like to handle this file with PowerShell.
Here is the part of my log file.
3 Spin_Up_Time 0x0020 100 100 000 Old_age Offline - 0
4 Start_Stop_Count 0x0030 100 100 000 Old_age Offline - 0
5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age Always - 0
And here is my code
$i = 1
$a = Get-Content log.txt
do {
$trimmed = $a[$i].trim()
$splitted = $trimmed.split(" ")
$i++
}while ($i -le 3)
If I use the .split(" ") it is working only with the thrid row.
How can I split my all rows correctly?
Thank you
A bit more code, but it gives you something that's a little easier to work with in the end:
$SMART = gc c:\temp\test.txt | %{
$temp = $_ -split " "|?{!([string]::IsNullOrWhiteSpace($_))}
new-object psobject -Property #{
"Entry"=$temp[0]
"TestName"=$temp[1]
"HexCode"=$temp[2]
"Number1"=$temp[3]
"Number2"=$temp[4]
"Number3"=$temp[5]
"Age"=$temp[6]
"Status"=$temp[7]
"Filler"=$temp[8]
"Zero?"=$temp[9]
}
}
$SMART|FT Entry,TestName,HexCode,Number1,Number2,Number3,Age,Status,Filler,Zero?
what does this do for you?
$a = Get-Content log.txt
-split $a
I get this
H:\> -split $a
3
Spin_Up_Time
0x0020
100
100
000
Old_age
Offline
-
0
4
Start_Stop_Count
0x0030
100
100
000
Old_age
Offline
-
0
5
Reallocated_Sector_Ct
0x0032
100
100
000
Old_age
Always
-
0
I like working with regex' here's a sample that allows you to name your columns.
$a = Get-Content log.txt
$pattern = [regex]'(?<rowid>\d+)\s(?<desc>[a-zA-Z_]+)\s+(?<hexdata>0x\d{4})\s+(?<col4>\d{3})\s+(?<col5>\d{3})\s+(?<col6>\d{3})\s+(?<text1>.+?)\s+(?<state>.+?)-\s+0'
foreach ($line in $a) {
if ($line -match $pattern) {
$dataobj = New-Object PSObject
$dataobj | Add-Member -type NoteProperty -name "Description" -value $matches['desc']
$dataobj | Add-Member -type NoteProperty -name "Hex Data" -value $matches['hexdata']
$dataobj | Add-Member -type NoteProperty -name "State" -value $matches['state']
$dataobj
}
}
Results:
Description Hex Data State
----------- -------- -----
Spin_Up_Time 0x0020 Offline
Start_Stop_Count 0x0030 Offline
Reallocated_Sector_Ct 0x0032 Always