Add Dynamic Key Value pair to PSCustomObject - powershell

i have a script which emits comma separated values for a particular variable and its always dynamic, how can i separate them into multiple key value pairs, i tried the below but the output array gives only values[0] and 1
$tagsd = #()
foreach ($a in $vms)
{
$tags = (($a.ExtensionData.GetResourceProperties()).property | ?{$_.name -eq 'summary|tag'}).value.TrimStart('[').TrimEnd(']').split(",").trim()
#<vSphere Tag-Production>, <Cost Center-90210>
$Object = [pscustomobject][ordered]#{
VmName = $a.name
}
0..$tags.count | % {
Add-Member -InputObject $Object -NotePropertyName "VMtag$_" -NotePropertyValue $tags[$_]
}
$tagsd += $Object
}
But the output comes as below, even export csv gives me the same
But the array consists of all the values as seen below, some VM's have Tag1 to Tag 5 but i get only VMtag[0] and VMtag1, can some one helpw?, i wanted all the tag values to be part of export csv.

the table is created according to the first object. If you want to display all the columns:
$propNames = $tagsd | foreach { $_.psobject.Properties.Name } | select -Unique
$tagsd | select $propNames
Also, it looks like there are other problems with your code.
If $tags has only one value (eg $tags = "none"), $tags[0] will return the first character "n" rather than the string "none". so you can work around this by either [array]$tags = ... or $tags = #(...).
In addition, the range should be 0..($tags.Count -1) instead of 0..$tags.Count.
$tagsd = #()
foreach ($a in $vms) {
[array]$tags = (($a.ExtensionData.GetResourceProperties()).Property | where { $_.Name -eq 'summary|tag' }).Value.TrimStart('[').TrimEnd(']').Split(",").Trim()
$object = [pscustomobject]#{ VmName = $a.Name }
if($tags.Count) {
0..($tags.Count - 1) | foreach {
$object | Add-Member -NotePropertyName "VMtag$_" -NotePropertyValue $tags[$_]
}
}
$tagsd += $object
}
If possible, it is better to avoid index access to array.
$tagsd = foreach ($a in $vms) {
$tags = (($a.ExtensionData.GetResourceProperties()).Property | where { $_.Name -eq 'summary|tag' }).Value.TrimStart('[').TrimEnd(']').Split(",").Trim()
$h = [ordered]#{ VmName = $a.Name }
$tags | foreach { $i = 0 } { $h.Add("VMtag${i}", $_); $i++ }
[pscustomobject]$h
}
$propNames = $tagsd | foreach { $_.psobject.Properties.Name } | select -Unique
$tagsd | select $propNames

Related

Check all lines in a huge CSV file in PowerShell

I want to work with a CSV file of more than 300,000 lines. I need to verify information line by line and then display it in a .txt file in the form of a table to see which file was missing for all servers. For example
Name,Server
File1,Server1
File2,Server1
File3,Server1
File1,Server2
File2,Server2
...
File345,Server76
File346,Server32
I want to display in table form this result which corresponds to the example above:
Name Server1 Server2 ... Server 32 ....Server 76
File1 X X
File2 X X
File3 X
...
File345 X
File346 X
To do this actually, I have a function that creates objects where the members are the Server Name (The number of members object can change) and I use stream reader to split data (I have more than 2 columns in my csv so 0 is for the Server name and 5 for the file name)
$stream = [System.IO.StreamReader]::new($File)
$stream.ReadLine() | Out-Null
while ((-not $stream.EndOfStream)) {
$line = $stream.ReadLine()
$strTempo = $null
$strTempo = $line -split ","
$index = $listOfFile.Name.IndexOf($strTempo[5])
if ($index -ne -1) {
$property = $strTempo[0].Replace("-", "_")
$listOfFile[$index].$property = "X"
}
else {
$obj = CreateEmptyObject ($listOfConfiguration)
$obj.Name = $strTempo[5]
$listOfFile.Add($obj) | Out-Null
}
}
When I export this I have a pretty good result. But the script take so much time (between 20min to 1hour)
I didn't know how optimize actually the script. I'm beginner to PowerShell.
Thanks for the futures tips
You might use HashSets for this:
$Servers = [System.Collections.Generic.HashSet[String]]::New()
$Files = #{}
Import-Csv -Path $Path |ForEach-Object {
$Null = $Servers.Add($_.Server)
if ($Files.Contains($_.Name)) { $Null = $Files[$_.Name].Add($_.Server) }
else { $Files[$_.Name] = [System.Collections.Generic.HashSet[String]]$_.Server }
}
$Table = foreach($Name in $Files.get_Keys()) {
$Properties = [Ordered]#{ Name = $Name }
ForEach ($Server in $Servers) {
$Properties[$Server] = if ($Files[$Name].Contains($Server)) { 'X' }
}
[PSCustomObject]$Properties
}
$Table |Format-Table -Property #{ expression='*' }
Note that in contrast to PowerShell's usual behavior, the .Net HashSet class is case-sensitive by default. To create an case-insensitive HashSet use the following constructor:
[System.Collections.Generic.HashSet[String]]::New([StringComparer]::OrdinalIgnoreCase)
See if this works faster. Change filename as required
$Path = "C:\temp\test1.txt"
$table = Import-Csv -Path $Path
$columnNames = $table | Select-Object -Property Server -Unique| foreach{$_.Server} | Sort-Object
Write-Host "names = " $columnNames
$groups = $table | Group-Object {$_.Name}
$outputTable = [System.Collections.ArrayList]#()
foreach($group in $groups)
{
Write-Host "Group = " $group.Name
$newRow = New-Object -TypeName psobject
$newRow | Add-Member -NotePropertyName Name -NotePropertyValue $group.Name
$servers = $group.Group | Select-Object -Property Server | foreach{$_.Server}
Write-Host "servers = " $servers
foreach($item in $columnNames)
{
if($servers.Contains($item))
{
$newRow | Add-Member -NotePropertyName $item -NotePropertyValue 'X'
}
else
{
#if you comment out next line code doesn't work
$newRow | Add-Member -NotePropertyName $item -NotePropertyValue ''
}
}
$outputTable.Add($newRow) | Out-Null
}
$outputTable | Format-Table

How could I add an if statement when combining two different arrays

today I asked
combined two arrays of different size with different properties
Here is the answer
$GroupMembersCount = $null
$GroupMembersCount = gam print groups domain <domain.com> members managers owners countsonly | ConvertFrom-Csv
$GroupSettings = $null
$GroupSettings = gam print groups settings | ConvertFrom-Csv
$GroupMemberCountByEmail = #{}
$GroupMembersCount | ForEach-Object {
$GroupMemberCountByEmail[$_.email] = $_
}
$GroupSettings | Select-Object *,
#{
Name = 'MembersCount'
Expression = { [int]$GroupMemberCountByEmail[$_.email].MembersCount }
},#{
Name = 'ManagersCount'
Expression = { [int]$GroupMemberCountByEmail[$_.email].ManagersCount }
},#{
Name = 'OwnersCount'
Expression = { [int]$GroupMemberCountByEmail[$_.email].OwnersCount }
} |
Export-Csv 'c:\temp\groups.csv' -NoTypeInformation -Encoding UTF8
How can I add a column to this answer above that will give me a True or false if all counts are zero ?
I think I need if before the export above but I am not sure how to If (X -eq 0 -and Y -eq 0) { Do stuff } here since I don't know how to address the X before the export.
| Export-Csv 'c:\temp\groups.csv' -NoTypeInformation -Encoding UTF8
I would just pipe to another Select-Object to keep your code cleaner. It is not required if you want to reuse your hash table lookups in another calculated property.
$GroupSettings | Select-Object *,
#{
Name = 'MembersCount'
Expression = { [int]$GroupMemberCountByEmail[$_.email].MembersCount }
},#{
Name = 'ManagersCount'
Expression = { [int]$GroupMemberCountByEmail[$_.email].ManagersCount }
},#{
Name = 'OwnersCount'
Expression = { [int]$GroupMemberCountByEmail[$_.email].OwnersCount }
} |
Select-Object *,
#{
Name='AllZeroes'
Expression={-not ($_.MembersCount -or $_.ManagersCount -or $_.OwnersCount) }
} |
Export-Csv 'c:\temp\groups.csv' -NoTypeInformation -Encoding UTF8
AllZeroes property will return $true if all counts are 0.
Again I want to add what I did. Please note to any reader some of my answer is different because I use a tool called GAM to help me administrate Google. GAM forced me to pivot from what I accepted above.
PS: I find with most answers here there are adjustments needed. Your mileage may vary so tread carefully.
<#
https://stackoverflow.com/questions/70613438/how-could-i-add-an-if-statement-when-combining-two-different-arrays/70614250#70614250
https://stackoverflow.com/questions/70609905/combined-two-arrays-of-different-size-with-different-properties/70610915?noredirect=1#comment124824384_70610915
#>
$GroupMembersCount = $null
$GroupMembersCount = gam print groups domain <Domain.com> members managers owners countsonly | ConvertFrom-Csv
$GroupSettings = $null
$GroupSettings = gam print groups settings | ConvertFrom-Csv
$GroupsEmpty = $null
$GroupsEmpty = gam config csv_output_row_filter "directMembersCount:count=0" print groups directmemberscount | ConvertFrom-csv
$GroupMemberCountByEmail = #{}
$GroupMembersCount | ForEach-Object {
$GroupMemberCountByEmail[$_.email] = $_
}
$Result = $null
$Result = $GroupSettings | Select-Object *,
#{
Name = 'MembersCount'
Expression = { [int]$GroupMemberCountByEmail[$_.email].MembersCount }
}, #{
Name = 'ManagersCount'
Expression = { [int]$GroupMemberCountByEmail[$_.email].ManagersCount }
}, #{
Name = 'OwnersCount'
Expression = { [int]$GroupMemberCountByEmail[$_.email].OwnersCount }
}
If ($GroupsEmpty) {
$Result | Add-Member -NotePropertyName EmptyGroup -NotePropertyValue $false
function Get-ArrayRowIndex {
param(
[parameter(mandatory = $true)][array]$Property,
[parameter(mandatory = $true)][string]$Value
)
[int]$index = 0
while ($index -lt ($Property.count)) {
if ($Property[$index] -eq $Value) {
return [int]$index
}
$index++
}
return $null
}
$Result | ForEach-Object {
If ($GroupsEmpty.email -contains $_.Email) {
$UpdateRowIndex = $null
#Get-ArrayRowIndex tries to find the item to update in the Result array. $UpdateRowIndex can't be [int]
$UpdateRowIndex = Get-ArrayRowIndex -Property $($Result.email) -Value $_.Email
#update the correct item in the array,changing flag made above.
If ($null -ne $UpdateRowIndex) {
$Result[$UpdateRowIndex].EmptyGroup = $true
}
Else {
#If location in array was not found function Get-ArrayRowIndex.
Write-Error "Problem with final report. Index location in the result array was not found using Get-ArrayRowIndex."
Start-Sleep -Seconds 3
break script
}
}
}
}
$result | Export-Csv 'c:\temp\groups.csv' -NoTypeInformation -Encoding UTF8
Try {
Invoke-Item -Path 'c:\temp\groups.csv'
}
Catch {
write-error "Final report not found at c:\temp\groups.csv"
}
There are better ways to do what I did above but this suits me for now.

How to add 2 columns in my PowerShell HTML report?

I have PowerShell script, I wrote this code:
$computers = Get-Content D:\Dev\Powershell\Powershell_TXT_FILE\pickup1.txt | Where {
-not ($_.StartsWith('#'))
} | foreach {
if (Test-Connection $_ -Quiet -Count 1) {
New-Object psobject -Property #{
Server = $_
Status = "Online"
}
} else {
New-Object PSObject -Property #{
Server = $_
Status = "Offline"
}
}
}
$computers | ConvertTo-Html -Property Server | Foreach {
if ($_ -like "*<td>Online</td>*" ) {
$_ -replace "<tr>","<tr bgcolor=green>"
} else {
$_ -replace "<tr>","<tr bgcolor=red>"
}
} | Out-File D:\Share\Powershell\Powershell_TXT_FILE\test.html
How to add 2 or 4 columns on script.
I want to get this conclusion:
Are you wanting columns 3 & 4 to exist but always be blank?
If so, try:
$computers = Get-Content D:\Dev\Powershell\Powershell_TXT_FILE\pickup1.txt | Where {
-not ($_.StartsWith('#'))
} | foreach {
if (Test-Connection $_ -Quiet -Count 1) {
New-Object psobject -Property #{
Server = $_
Status = "Online"
}
} else {
New-Object PSObject -Property #{
Server = $_
Status = "Offline"
}
}
}
$computers | ConvertTo-Html -Property Server | Foreach {
if ($_ -like "*<td>Online</td>*" ) {
$_ -replace "<tr>","<tr bgcolor=green>"
} else {
$_ -replace "<tr>","<tr bgcolor=red>"
} | %{$_ -replace "</td></tr>","</td><td> </td><td> </td></tr>"}
} | Out-File D:\Share\Powershell\Powershell_TXT_FILE\test.html
This adds the two columns with a space in them, to the end of each row in your table. (based on the value in the HTML table.
Now, if you don't mind headers, it easier, add to the first line the data fields you want..
$computers = Get-Content D:\Dev\Powershell\Powershell_TXT_FILE\pickup1.txt | Where { -not ($_.StartsWith('#')) | select *,xx,yy
where XX and YY are your two added fields in your HTML table.

Converting CSV Column to Rows

I'm trying to create a PowerShell script that transpose a CSV file from column to rows.
I found examples of doing the opposite (converting row based CSV to column) but I found nothing on column to rows. My problem being that I don't know exactly how many column I'll have. I tried adapting the row to column to column to rows but unsuccessfully.
$a = Import-Csv "input.csv"
$a | FT -AutoSize
$b = #()
foreach ($Property in $a.Property | Select -Unique) {
$Props = [ordered]#{ Property = $Property }
foreach ($Server in $a.Server | Select -Unique){
$Value = ($a.where({ $_.Server -eq $Server -and
$_.Property -eq $Property })).Value
$Props += #{ $Server = $Value }
}
$b += New-Object -TypeName PSObject -Property $Props
}
$b | FT -AutoSize
$b | Out-GridView
$b | Export-Csv "output.csv" -NoTypeInformation
For example my CSV can look like this:
"ID","DATA1"
"12345","11111"
"54321","11111"
"23456","44444"
or this (number of column can vary):
"ID","DATA1","DATA2","DATA3"
"12345","11111","22222","33333"
"54321","11111",,
"23456","44444","55555",
and I would like the script to convert it like this:
"ID","DATA"
"12345","11111"
"12345","22222"
"12345","33333"
"54321","11111"
"23456","44444"
"23456","55555"
The trick is to query the members of the table to get the column names. Once you do that then the rest is straightforward:
function Flip-Table ($Table) {
Process {
$Row = $_
# Get all the columns names, excluding the ID field.
$Columns = ($Row | Get-Member -Type NoteProperty | Where-Object Name -ne ID).Name
foreach ($Column in $Columns) {
if ($Row.$Column) {
$Properties = [Ordered] #{
"ID" = $Row.ID
"DATA" = $Row.$Column
}
New-Object PSObject -Property $Properties
}
}
# Garbage collection won't kick in until the end of the script, so
# invoke it every 100 input rows.
$Count++;
if (($Count % 100) -eq 0) {
[System.GC]::GetTotalMemory('forceFullCollection') | out-null
}
}
}
Import-Csv input.csv | Flip-Table | Export-Csv -NoTypeInformation output.csv
Well, here is mine. I'm not as fancy as the rest:
$in = Get-Content input.csv | Select -Skip 1
$out = New-Object System.Collections.ArrayList
foreach($row in $in){
$parts = $row.Split(',')
$id = $parts[0]
foreach($data in $parts[1..$parts.Count]){
if($data -ne '' -AND $data -ne $null){
$temp = New-Object PSCustomObject -Property #{'ID' = $id;
'Data' = $data}
$out.Add($temp) | Out-Null
}
}
}
$out | Export-CSV output.csv -NoTypeInformation
You can do something like this
# Convert csv to object
$csv = ConvertFrom-Csv #"
"ID","DATA1","DATA2","DATA3"
"12345","11111","22222","33333"
"54321","11111",,
"23456","44444","55555"
"#
# Ignore common members and the ID property
$excludedMembers = #(
'GetHashCode',
'GetType',
'ToString',
'Equals',
'ID'
)
$results = #()
# Iterate around each csv row
foreach ($row in $csv) {
$members = $row | Get-Member
# Iterate around each member from the 'row object' apart from our
# exclusions and empty values
foreach ($member in $members |
Where { $excludedMembers -notcontains $_.Name -and $row.($_.Name)}) {
# add to array of objects
$results += #{ ID=$row.ID; DATA=$row.($member.Name)}
}
}
# Write the csv string
$outstring = "ID,DATA"
$results | foreach { $outstring += "`n$($_.ID),$($_.DATA)" }
# New csv object
$csv = $outstring | ConvertFrom-Csv
Probably not the most elegant solution, but should do what you need
I left some comments explaining what it does
If you only want to accept a limited number DATA columns (e.g. 5), you could do:
ForEach ($i in 1..5) {$CSV | ? {$_."Data$i"} | Select ID, #{N='Data'; E={$_."Data$i"}}}
And if you have a potential unlimited number of DATA columns:
ForEach ($Data in ($CSV | Select "Data*" -First 1).PSObject.Properties.Name) {
$CSV | ? {$_.$Data} | Select ID, #{N='Data'; E={$_.$Data}}
}

Filter csv file by array of hashtables

What I am trying todo is filter a csv file (happens to be a weblog) with an array of hashtables (user information from a database).
#$data is from a database. (about 500 items) Type: System.Data.DataTable
$users = #()
foreach($row in $data)
{
$userItem = #{
LoginId = $row[0]
LastName = $row[3]
FirstName = $row[4]
LastAccess = $null
}
$users += $userItem
}
#Log files are about 14,000 lines long
$logfiles = Get-ChildItem $logFolder -Recurse | where {$_.Extension -eq ".log"} | Sort-Object BaseName -Descending
foreach($log in $logfiles)
{
$csvLog = Import-Csv $log.FullName -Header ("Blank","LoginId","Date")
$u = $users | Select {&_.LoginId}
$filteredcsvLog = $cvsLog | Where-Object { $u -contains $_.LoginId}
#This returns null
....
}
This does not seem to work, what am I missing. My guess is that I need to flatten the array into [string[]], however I can't seem todo that either.
Rather than do an array of hashtables, I would do a hashtable of custom objects e.g.:
$users = #{}
foreach($row in $data)
{
$userItem = new-object psobject -property #{
LoginId = $row[0]
LastName = $row[3]
FirstName = $row[4]
LastAccess = $null
}
$users[$userItem.LoginId] = $userItem
}
Then the filtering is easier and faster:
foreach($log in $logfiles)
{
$csvLog = Import-Csv $log.FullName -Header ("Blank","LoginId","Date")
$filteredcsvLog = $cvsLog | Where-Object { $users[$_.LoginId} }
....
}