I keep running into the same problem again, and i have my default way of handling it, but it keeps bugging me.
Isn't there any better way?
So basicly i have a pipline running, do stuff within the pipline, and want to return a Key/Value Pair from within the pipline.
I want the whole pipline to return a object of type psobject (or pscustomobject).
Here is the way i do it everytime.
I create a hashtable at the beginning of the pipline and add key/Value Pairs from within the pipline to this hashtable using the .Add() method.
Afterwards i create a psobject by passing the hashtbale to New-Object`s -Property Parameter. This gives me the desired result.
Get-Process | Sort -Unique Name | ForEach-Object -Begin { $ht = #{} } -Process {
# DO STUFF
$key = $_.Name
$val = $_.Id
# Add Entry to Hashtable
$ht.Add($key,$val)
}
# Create PSObject from Hashtable
$myAwesomeNewObject = New-Object psobject -Property $ht
# Done - returns System.Management.Automation.PSCustomObject
$myAwesomeNewObject.GetType().FullName
But this seems a bit cluncky, isn't there a more elegant way of doing it?
Something like this:
[PSObject]$myAwesomeNewObject = Get-Process | Sort -Unique Name | ForEach-Object -Process {
# DO STUFF
$key = $_.Name
$val = $_.Id
# return Key/Val Pair
#{$key=$val}
}
# Failed - returns System.Object[]
$myAwesomeNewObject.GetType().FullName
This unfortunally dosn't work, since the pipe returns an array of hashtables, but i hope you know now what iam trying to achieve.
Thanks
Not sure if this is more elegant but just another way of doing it, this uses an anonymous function so $ht will no longer be available after execution, and casts to [pscustomobject] instead of using New-Object:
[pscustomobject] (Get-Process | Sort -Unique Name | & {
begin { $ht = #{ } }
process {
# DO STUFF
$key = $_.Name
$val = $_.Id
# Add Entry to Hashtable
$ht.Add($key, $val)
}
end { $ht }
})
You can also use the -End parameter to convert the final hash table to a pscustomobject as part of the pipeline, without needing to set the whole thing to a variable
$ht[$key]=$val is also a nice shorthand for $ht.Add($key,$val):
Get-Process |
Sort -Unique Name |
Foreach -Begin { $ht = #{} } -Process {
$ht[$_.Name] = $_.Id
} -End {[pscustomobject]$ht} |
## continue pipeline with pscustomobject
Thanks to #Santiago Squarzon and #Cpt.Whale answers, i were able to combine them to create a solution that pleases me:
$myAwesomeNewObject = `
Get-Process | Sort -Unique Name | & {
begin { $ht = #{} }
process {
# DO STUFF
$key = $_.Name
$val = $_.Id
# Add Entry to Hashtable
$ht[$key]=$val
}
end {[pscustomobject]$ht}
}
# Success - System.Management.Automation.PSCustomObject
$myAwesomeNewObject.Gettype().FullName
# And helper Hashtable is NULL thanks to the
# anonym function
$null -eq $ht
Thanks alot Guys
Alternatively you may create a hashtable using Group-Object -AsHashTable:
# Store the PIDs of all processes into a PSCustomObject, keyed by the process name
$processes = [PSCustomObject] (Get-Process -PV proc |
Select-Object -Expand Id |
Group-Object { $proc.Name } -AsHashtable)
# List all PIDs of given process
$processes.chrome
Notes:
Common parameter -PV (alias of -PipelineVariable) makes sure that we can still access the full process object from within the calculated property of the Group-Object command, despite that we have a Select-Object command in between.
The values of the properties are arrays, which store the process IDs of all instances of each process. E. g. $processes.chrome outputs a list of PIDs of all instances of the chrome process.
I am trying to find an elegant way to put the metadata of a table of type System.Data.DataTable into a multi-dimensional array for easy reference in my program. My approach to the issue so far seems tedious.
Assuming $DataTable being the DataTable in question
What I tried to do so far was:
$Types = $DataTable.Columns | Select-Object -Property DataType
$Columns= $DataTable.Columns | Select-Object -Property ColumnName
$Index = $DataTable.Columns | Select-Object -Property ordinal
$AllowNull = $DataTable.Columns | Select-Object -Property AllowDbNull
Then painfully going through each array, pick up individual items and put them in my multi-dimensional array $TableMetaData.
I read in the documentation of Select-Object and it seems to me that only 1 property can be selected at 1 time? I think I should be able to do all the above more elegantly and store the information in $TableMetaData.
Is there a way to easily pick up multiple properties and put them in a multi-dimensional array in 1 swoop?
I read the documentation of Select-Object and it seems to me that only 1 property can be selected at 1 time?
This is not true, Select-Object can take any number of arguments to the -Property parameter
$ColumnInfo = $DataTable.Columns | Select-Object -Property DataType,ColumnName,ordinal,AllowDbNull
Now $ColumnInfo will contain one object for each column, having all 4 properties.
Rather than using a multi-dimensional array, you should consider using a hashtable (#{}, an unordered dictionary):
$ColumnInfo = $DataTable.Columns | ForEach-Object -Begin { $ht = #{} } -Process {
$ht[$_.ColumnName] = $_
} -End { return $ht }
Here, we create an empty hashtable $ht (the -Begin block runs just once), then store each column object in $ht using the ColumnName as the key, and finally return $ht, storing it in $ColumnInfo.
Now you can reference metadata about each column by Name:
$ColumnInfo.Column2
# or
$ColumnInfo["Column2"]
One easy way to do this is to create an "empty" variable with Select-Object. Here is a sample command:
$DataTableReport = "" | Select-Object -Property DataType, ColumnName, ordinal, AllowDbNull
Then, link the $DataTableReport to the $Types, $Columns, $Index, and the $AllowNull properties as shown below:
$DataTableReport.Types = $DataTable.DataType
$DataTableReport.Columns = $DataTable.ColumnName
$DataTableReport.Index = $DataTable.ordinal
$DataTableReport.AllowNull = $DataTable.AllowDbNull
Finally, call the DataTableReport variable.
$DataTableReport # will display all the results in a tabular form.
I don't have much experience with CSV, so apologies if I'm really blind here.
I have a basic CSV and script setup to test this with. The CSV has two columns, Letter and Number. Letter goes from A-F and Number goes from 1-10. This means that Number has more rows than Letter, so when running the following script, the output can sometimes provide an empty Letter.
$L = ipcsv ln.csv | Get-Random | Select-Object -ExpandProperty Letter
$N = ipcsv ln.csv | Get-Random | Select-Object -ExpandProperty Number
Write-Output $L
Write-Output $N
Some outputs come out as
B
9
while others can come out as
5
I don't know whether the issue is my script not ignoring empty lines or my CSV being written incorrectly, which is posted below.
Letter,Number
A,1
B,2
C,3
D,4
E,5
F,6
,7
,8
,9
,10
What's my issue here and how do I go about fixing it?
Your asking for a random object from your CSV, not a random letter. Since some of the lines are missing a letter, you might end up picking one that has an empty Letter-value.
If you want to pick any line with a letter, you need to filter the rows first to only pick from the ones with a value. Also, you sould avoid reading the same file twice, use a varible
#$csv = Import-CSV -Path ln.csv
$csv = #"
Letter,Number
A,1
B,2
C,3
D,4
E,5
F,6
,7
,8
,9
,10
"# | ConvertFrom-Csv
$L = $csv | Where-Object { $_.Letter } | Get-Random | Select-Object -ExpandProperty Letter
$N = $csv | Where-Object { $_.Number } | Get-Random | Select-Object -ExpandProperty Number
Write-Output $L
Write-Output $N
CSV migtht not be the best solution for this scenario. Ex. you could store these as arrays in the script, like:
$chars = [char[]](65..70) #A-F uppercase letters
$numbers = 1..10
$L = $chars | Get-Random
$N = $numbers | Get-Random
Write-Output $L
Write-Output $N
Import-Csv turns each line into an object, with a property for each column.
Even though one or more property values may be empty, the object still exists, and Get-Random has no reason determine that an object with a certain property (such as Letter) having the value "" (ie. an empty string), should not be picked.
You can fix this by expanding the property values first, then filter for empty values and then finally pick the random value from those that weren't empty:
$L = ipcsv ln.csv |Select-Object -ExpandProperty Letter |Where-Object {$_} |Get-Random
$N = ipcsv ln.csv |Select-Object -ExpandProperty Number |Where-Object {$_} |Get-Random
I'm newbie in Powershell. I tried to process / transpose row-column against a medium size csv based record (around 10000 rows). The original CSV consist of around 10000 rows with 3 columns ("Time","Id","IOT") as below:
"Time","Id","IOT"
"00:03:56","23","26"
"00:03:56","24","0"
"00:03:56","25","0"
"00:03:56","26","1"
"00:03:56","27","0"
"00:03:56","28","0"
"00:03:56","29","0"
"00:03:56","30","1953"
"00:03:56","31","22"
"00:03:56","32","39"
"00:03:56","33","8"
"00:03:56","34","5"
"00:03:56","35","269"
"00:03:56","36","5"
"00:03:56","37","0"
"00:03:56","38","0"
"00:03:56","39","0"
"00:03:56","40","1251"
"00:03:56","41","103"
"00:03:56","42","0"
"00:03:56","43","0"
"00:03:56","44","0"
"00:03:56","45","0"
"00:03:56","46","38"
"00:03:56","47","14"
"00:03:56","48","0"
"00:03:56","49","0"
"00:03:56","2013","0"
"00:03:56","2378","0"
"00:03:56","2380","32"
"00:03:56","2758","0"
"00:03:56","3127","0"
"00:03:56","3128","0"
"00:09:16","23","22"
"00:09:16","24","0"
"00:09:16","25","0"
"00:09:16","26","2"
"00:09:16","27","0"
"00:09:16","28","0"
"00:09:16","29","21"
"00:09:16","30","48"
"00:09:16","31","0"
"00:09:16","32","4"
"00:09:16","33","4"
"00:09:16","34","7"
"00:09:16","35","382"
"00:09:16","36","12"
"00:09:16","37","0"
"00:09:16","38","0"
"00:09:16","39","0"
"00:09:16","40","1882"
"00:09:16","41","42"
"00:09:16","42","0"
"00:09:16","43","3"
"00:09:16","44","0"
"00:09:16","45","0"
"00:09:16","46","24"
"00:09:16","47","22"
"00:09:16","48","0"
"00:09:16","49","0"
"00:09:16","2013","0"
"00:09:16","2378","0"
"00:09:16","2380","19"
"00:09:16","2758","0"
"00:09:16","3127","0"
"00:09:16","3128","0"
...
...
...
I tried to do the transpose using code based from powershell script downloaded from https://gallery.technet.microsoft.com/scriptcenter/Powershell-Script-to-7c8368be
Basically my powershell code is as below:
$b = #()
foreach ($Time in $a.Time | Select -Unique) {
$Props = [ordered]#{ Time = $time }
foreach ($Id in $a.Id | Select -Unique){
$IOT = ($a.where({ $_.Id -eq $Id -and $_.time -eq $time })).IOT
$Props += #{ $Id = $IOT }
}
$b += New-Object -TypeName PSObject -Property $Props
}
$b | FT -AutoSize
$b | Out-GridView
Above code could give me the result as I expected which are all "Id" values will become column headers while all "Time" values will become unique row and "IOT" values as the intersection from "Id" x "Time" as below:
"Time","23","24","25","26","27","28","29","30","31","32","33","34","35","36","37","38","39","40","41","42","43","44","45","46","47","48","49","2013","2378","2380","2758","3127","3128"
"00:03:56","26","0","0","1","0","0","0","1953","22","39","8","5","269","5","0","0","0","1251","103","0","0","0","0","38","14","0","0","0","0","32","0","0","0"
"00:09:16","22","0","0","2","0","0","21","48","0","4","4","7","382","12","0","0","0","1882","42","0","3","0","0","24","22","0","0","0","0","19","0","0","0"
While it only involves a few hundreds rows, the result comes out quickly as expected, but the problem now when processing the whole csv file with 10000 rows, the script above 'keep executing' and doesn't seem able to finish for long time (hours) and couldn't spit out any results.
So probably if some powershell experts from stackoverflow could help to asses the code above and probably could help to modify to speed up the results?
Many thanks for the advise
10000 records is a lot but I don't think it is enough to advise streamreader* and manually parsing the CSV. The biggest thing going against you though is the following line:
$b += New-Object -TypeName PSObject -Property $Props
What PowerShell is doing here is making a new array and appending that element to it. This is a very memory intensive operation that you are repeating 1000's of times. Better thing to do in this case is use the pipeline to your advantage.
$data = Import-Csv -Path "D:\temp\data.csv"
$headers = $data.ID | Sort-Object {[int]$_} -Unique
$data | Group-Object Time | ForEach-Object{
$props = [ordered]#{Time = $_.Name}
foreach($header in $headers){
$props."$header" = ($_.Group | Where-Object{$_.ID -eq $header}).IOT
}
[pscustomobject]$props
} | export-csv d:\temp\testing.csv -NoTypeInformation
$data will be your entire file in memory as an object. Need to get all the $headers that will be the column headers.
Group the data by each Time. Then inside each time object we get the value for every ID. If the ID does not exist during that time then the entry will show as null.
This is not the best way but should be faster than yours. I ran 10000 records in under a minute (51 second average over 3 passes). Will benchmark to show you if I can.
I just ran your code once with my own data and it took 13 minutes. I think it is safe to say that mine performs faster.
Dummy data was made with this logic FYI
1..100 | %{
$time = get-date -Format "hh:mm:ss"
sleep -Seconds 1
1..100 | % {
[pscustomobject][ordered]#{
time = $time
id = $_
iot = Get-Random -Minimum 0 -Maximum 7
}
}
} | Export-Csv d:\temp\data.csv -notypeinformation
* Not a stellar example for your case of streamreader. Just pointing it out to show that it is the better way to read large files. Just need to parse string line by line.