Powershell custom object table manipulation - powershell

I have a PS object with name, column2, column3, etc. I have 3 names and values in each of the columns for those names. I want to loop through and put into a table name1, column1, name2, column1, name3, column1, name1, column2, name2, column2, name3, column2, name1, column3, name2, column3, name3, column3, etc. is there a way to do this?
I have tried and object for names and one for columns and using a for each loop. I'm trying to automate lines of code instead of using a select object to select each name and column.
current code:
$obj | sort-object Name | Select-object AER, DER, APRO,FPCT
New-HTML {
New-HTMLSection -Invisible {
New-HTMLSection {
New-HTMLTable -DataTable $obj -includeproperty Name, AER -DefaultSortOrder Descending -PagingLength 10
}
New-HTMLSection {
New-HTMLTable -DataTable $obj -includeproperty Name, DER -DefaultSortOrder Descending -PagingLength 10
}
New-HTMLSection {
New-HTMLTable -DataTable $obj -includeproperty Name, APRO -DefaultSortOrder Descending -PagingLength 10
}
New-HTMLSection {
New-HTMLTable -DataTable $obj -includeproperty Name, FPCT -DefaultSortOrder Descending -PagingLength 10
}
}
Was looking to do something like this:
New-HTML {
New-HTMLSection {
$Machines = #(
'Machine1', 'Machine2', 'Machine3'
)
foreach ($Machine in $Machines) {
New-HTMLSection -HeaderText $Machine {
$Information = #(
'First Information', '2nd Information', '3rd information'
)
foreach ($I in $Information) {
New-HTMLSection -HeaderText $I {
New-HTMLTable -DataTable $Machine
}
}
} -Direction column
}
} -HeaderText "Citrix Machine Information" -Direction column
} -Online -ShowHTML
current and desired output:

Related

ForEach Nested Loop in PowerShell

I have this piece of code where I am extracting table names from the adventureworks.bim file using a for each loop. However, I am missing something here because I required a Table name per object and not the final table in the loop. The details are as below
cls
$BIM = "C:\Users\Desktop\adventureworks.bim"
$origmodel = (Get-Content $BIM -Raw) | Out-String | ConvertFrom-Json
ForEach($table in $origmodel.Model.tables.name)
{
$ColumnProperty = $origmodel.Model.tables.columns | ForEach-Object {
[pscustomobject] #{
'Table Name' = $table
'Object Name' = $_.name
'DataType' = $_.dataType
}
}
}
$ColumnProperty | ConvertTo-Csv -NoTypeInformation
Result I Get
"Table Name", "Object Name", "Datatype"
"Date","RowNumber-2662979B-1795-4F74-8F37-6A1BA8059B61","int64"
"Date","CurrencyKey","int64"
"Date","Currency Code","string"
"Date","CurrencyName","string"
"Date","RowNumber-2662979B-1795-4F74-8F37-6A1BA8059B61","int64"
"Date","CustomerKey","int64"
"Date","GeographyKey","int64"
"Date","Customer Id","string"
"Date","Title","string"
"Date","First Name","string"
"Date","Middle Name","string"
"Date","Last Name","string"
"Date","Name Style","boolean",
"Date","Birth Date","dateTime",
"Date","Marital Status","string"
"Date","Suffix","string"
"Date","Gender","string"
"Date","RowNumber-2662979B-1795-4F74-8F37-6A1BA8059B61","int64"
"Date","DateKey","int64"
"Date","Date","dateTime",
"Date","Day Number Of Week","int64"
"Date","Day Name Of Week","string"
"Date","Day Of Year","int64"
"Date","Week Of Year","int64"
"Date","Month Name","string"
Result I Need
"Table Name", "Object Name", "DataType"
"Currency","RowNumber-2662979B-1795-4F74-8F37-6A1BA8059B61","int64"
"Currency","CurrencyKey","int64"
"Currency","Currency Code","string"
"Currency","CurrencyName","string"
"Customer","RowNumber-2662979B-1795-4F74-8F37-6A1BA8059B61","int64"
"Customer","CustomerKey","int64"
"Customer","GeographyKey","int64"
"Customer","Customer Id","string"
"Customer","Title","string"
"Customer","First Name","string"
"Customer","Middle Name","string"
"Customer","Last Name","string"
"Customer","Name Style","boolean",
"Customer","Birth Date","dateTime",
"Customer","Marital Status","string"
"Customer","Suffix","string"
"Customer","Gender","string"
"Date","RowNumber-2662979B-1795-4F74-8F37-6A1BA8059B61","int64"
"Date","DateKey","int64"
"Date","Date","dateTime",
"Date","Day Number Of Week","int64"
"Date","Day Name Of Week","string"
"Date","Day Of Year","int64"
"Date","Week Of Year","int64"
"Date","Month Name","string"
Try this on for size:
$BIM = "C:\Users\Desktop\adventureworks.bim"
$origmodel = Get-Content $BIM -Raw | ConvertFrom-Json
ForEach ($table in $origmodel.Model.tables) {
$ColumnProperty += $table.columns | ForEach-Object {
[pscustomobject] #{
'Table Name' = $table.name
'Object Name' = $_.name
'DataType' = $_.dataType
}
}
}
$ColumnProperty | ConvertTo-Csv -NoTypeInformation
Corrections
No need to use Out-String
ColumnProperty needed a += as = was overwriting every entry
Just a few mix-ups around the foreach loops.
You need to loop over .model.tables first then an inner loop for each columns:
$req = Invoke-RestMethod https://raw.githubusercontent.com/TabularEditor/TabularEditor/master/TabularEditorTest/AdventureWorks.bim
$req.model.tables | ForEach-Object {
foreach($column in $_.columns) {
[pscustomobject]#{
Table = $_.name
ObjectName = $column.name
DataType = $column.dataType
}
}
} | Export-Csv path\to\export.csv -NoTypeInformation
Output to the console would look like this for the first few objects:
Table ObjectName DataType
----- ---------- --------
Currency RowNumber-2662979B-1795-4F74-8F37-6A1BA8059B61 int64
Currency CurrencyKey int64
Currency Currency Code string
Currency CurrencyName string
Customer RowNumber-2662979B-1795-4F74-8F37-6A1BA8059B61 int64
Customer CustomerKey int64
Customer GeographyKey int64
Customer Customer Id string
Customer Title string
...
...

Problem with exporting and appending data to csv file

The situation is the following - I've got a csv file with number of columns (let's say 3) and custom object with same columns but one will be a new one. I want to compare column names and if there is a new one, then add column and then add values accordingly to csv file and I don't want to replace all existing content in current csv file. Here is the code I've written, but I'm getting error: The appended object does not have a property that c
orresponds to the following column: column1. To continue with mismatched properties, add the -Force parameter, and then retry the command.
File sample:
Output should be same file, but with new column 'column4' and new row containing, like this:
I can't figure out what I'm doing wrong.
$file_path = '..path_to_the_file\test_csv.csv'
$csv = import-csv -path $file_path
$columns = #([pscustomobject]#{
column1 = 'something_new_1'
column2 = 'something_new_2'
column3 = 'something_new_3'
column4 = 'something_new_4'
}
)
$csv_columns = ($csv | get-member).where({$_.Membertype -eq 'NoteProperty'}).Name
$columns = ($columns | get-member).where({$_.Membertype -eq 'NoteProperty'}).Name
$compare = $columns | Where-Object {$csv_columns -notContains $_}
foreach ($column in $csv) {
$column | add-member -MemberType NoteProperty -Name $compare -value ''
}
$csv | export-csv -Path $file_path -NoTypeInformation
$columns | export-csv -path $file_path -Append -NoTypeInformation -force
You can use this function to streamline the process, this of course assumes the objects you want to append already has the same properties as the ones in the array of objects and will only add new ones.
function Add-ObjectNormalized {
[CmdletBinding()]
param(
[Parameter(Mandatory, ValueFromPipeline)]
[object] $InputObject,
[Parameter(Mandatory, Position = 0)]
[object[]] $AddObject
)
begin {
$isFirstObject = $false
$propertiesToAdd = [Collections.Generic.HashSet[string]]::new(
[string[]] $AddObject[0].PSObject.Properties.Name,
[System.StringComparer]::OrdinalIgnoreCase
)
}
process {
if(-not $isFirstObject) {
$isFirstObject = $true
$propertiesToAdd.ExceptWith([string[]] $InputObject.PSObject.Properties.Name)
}
$psobject = $InputObject.PSObject.Properties
foreach($prop in $propertiesToAdd) {
$psobject.Add([psnoteproperty]::new($prop, $null))
}
$InputObject
}
end {
$AddObject
}
}
Pipe it after Import-Csv including the array of objects to be added as argument. For example:
$objects = #(
[pscustomobject]#{
column1 = 'something_new_1'
column2 = 'something_new_2'
column3 = 'something_new_3'
column4 = 'something_new_4'
}
[pscustomobject]#{
column1 = 'something_new_5'
column2 = 'something_new_6'
column3 = 'something_new_7'
column4 = 'something_new_8'
}
)
Import-Csv path\to\csv.csv | Add-ObjectNormalized $objects |
Export-Csv path\to\newCsv.csv -NoTypeInformation
The output using a simple Csv like the one in question would be:
column1 column2 column3 column4
------- ------- ------- -------
va1 val2 val3
something_new_1 something_new_2 something_new_3 something_new_4
something_new_5 something_new_6 something_new_7 something_new_8

Compare-Object to Return All Rows in Array Which Have the Same Name

This compare call returns all rows that have the same Name equal in both Tables - But the other fields, like the DateField are returning no data and they have date data.
How can I correctly use the compare to return all the data from these rows that have the same Name values?
birth_date FirstName object_id
blank John blank
blank David blank
$otherField1 = "birth_date"
$otherField2 = "object_id"
$Name = "FirstName"
$exportObject=#()
$exportObject = Compare-Object -ReferenceObject $1_resultsDataTable -DifferenceObject $1_resultsDataTable -IncludeEqual -Property $Name| Where-Object{($_.SideIndicator -eq '==') } | Select-object $otherField1, $Name, $otherField2
Given you are comparing complex objects. I don't think Compare-Object is the right tool. When using the -Property parameter Compare-Object will strip the original input objects from it's output objects. -PassThru will amend the input objects with the SideIndicator property, but it will not include data from both the objects for whom the given property was equivalent. In cases where other property values differ it will not be represented in the output.
$ObjectArray1 =
#(
[PSCustomObject]#{
Birth_Date = [DateTime]'4/22/65'
FirstName = "Otto"
Object_id = 1234
}
[PSCustomObject]#{
Birth_Date = [DateTime]'5/30/67'
FirstName = "Jean"
Object_id = 9999
}
)
$ObjectArray2 =
#(
[PSCustomObject]#{
Birth_Date = [DateTime]'4/1/80'
FirstName = "Bob"
Object_id = 1234
}
[PSCustomObject]#{
Birth_Date = [DateTime]'5/30/67'
FirstName = "Jean"
Object_id = 9998
Prop4 = "Some other val"
}
)
Compare-Object $ObjectArray1 $ObjectArray2 -IncludeEqual -ExcludeDifferent -Property FirstName -PassThru
This returns
Birth_Date FirstName Object_id SideIndicator
---------- --------- --------- -------------
5/30/1967 12:00:00 AM Jean 9999 ==
Notice there's only 1 object. There's no indication of Prop4, nor that the second object had a different Object_id. Only the reference object is output.
Something like below would reveal such differences.
$Output = [Collections.ArrayList]#()
$ObjectArray1 |
ForEach-Object{
If( $_.FirstName -in $ObjectArray2.FirstName ) {
[Void]$Output.Add($_)
}
}
$ObjectArray2 |
ForEach-Object{
If( $_.FirstName -in $ObjectArray1.FirstName ) {
[Void]$Output.Add($_)
}
}
$Output
Output:
Birth_Date FirstName Object_id
---------- --------- ---------
5/30/1967 12:00:00 AM Jean 9999
5/30/1967 12:00:00 AM Jean 9998
I suppose it depends on the nature of the objects and tables. However, it strikes me that operators like -contains & -on it just strikes me that are better options.
It would be even easier if when the first name is the same all other properties are the same. Essentially you would only need to output from one of the collections:
$ObjectArray1 =
#(
#...
[PSCustomObject]#{
Birth_Date = [DateTime]'5/30/67'
FirstName = "Jean"
Object_id = 9999
}
)
$ObjectArray2 =
#(
# ...
[PSCustomObject]#{
Birth_Date = [DateTime]'5/30/67'
FirstName = "Jean"
Object_id = 9999
}
)
$ObjectArray1 |
Where-Object{ $_.FirstName -in $ObjectArray2.FirstName }

Handling multiple CSVs in Powershell efficiently

I am retrieving two CSVs from an API, one called students.csv similar to:
StudentNo,PreferredFirstnames,PreferredSurname,UPN
111, john, smith, john#email.com
222, jane, doe, jane#email.com
one called rooms.csv:
roomName, roomNo, students
room1, 1, {#{StudentNo=111; StudentName=john smith; StartDate=2018-01-01T00:00:00; EndDate=2018-07-06T00:00:00},....
room2, 2,{#{StudentNo=222; StudentName=jane doe; StartDate=2018-01-01T00:00:00; EndDate=2018-07-06T00:00:00},...
The third column in rooms.csv is an array as provided by the API
What is the best way to consolidate the two into
StudentNo,PreferredFirstnames,PreferredSurname,UPN, roomName
111, john, smith, john#email.com, room1
222, jane, doe, jane#email.com, room2
Im thinking something like...
$rooms = Import-Csv rooms.csv
$students = Import-Csv students.csv
$combined = $students | select-object StudentNo,PreferredSurname,PreferredFirstnames,UPN,
#{Name="roomName";Expression={ ForEach ($r in $rooms) {
if ($r.Students.StudentNo.Contains($_.StudentNo) -eq "True")
{return $r.roomName}}}}
This works, but is the foreach the right way to go am i mixing things up or is there a more efficient way???
--- Original Post ---
With all of this information I need to compare the student data and update AzureAD and then compile a list of data including first name, last name, upn, room and others that are retrieved from AzureAD.
My issue is "efficiency". I have code that mostly works but it takes hours to run. Currently I am looping through students.csv and then for each student looping through rooms.csv to find the room they're in, and obviously waiting for multiple api calls in-between all this.
What is the most efficient way to find the room for each student? Is importing the CSV as a custom PSObject comparable to using hash tables?
I was able to get your proposed code to work but it requires some tweaks to the code and data:
There must be some additional step where you are deserializing the students column of rooms.csv to a collection of objects. It appears to be a ScriptBlock that evaluates to an array of HashTables, but some changes to the CSV input are still needed:
The StartDate and EndDate properties need to be quoted and cast to [DateTime].
At least for rooms that contain multiple students, the value must be quoted so Import-Csv doesn't interpret the , separating array elements as an additional column.
The downside of using CSV as an intermediate format is the original property types are lost; everything becomes a [String] upon import. Sometimes it's desirable to cast back to the original type for efficiency purposes, and sometimes it's absolutely necessary in order for certain operations to work. You could cast those properties every time you use them, but I prefer to cast them once immediately after import.
With those changes rooms.csv becomes...
roomName, roomNo, students
room1, 1, "{#{StudentNo=111; StudentName='john smith'; StartDate=[DateTime] '2018-01-01T00:00:00'; EndDate=[DateTime] '2018-07-06T00:00:00'}}"
room2, 2, "{#{StudentNo=222; StudentName='jane doe'; StartDate=[DateTime] '2018-01-01T00:00:00'; EndDate=[DateTime] '2018-07-06T00:00:00'}}"
...and the script becomes...
# Replace the [String] property "students" with an array of [HashTable] property "Students"
$rooms = Import-Csv rooms.csv `
| Select-Object `
-ExcludeProperty 'students' `
-Property '*', #{
Name = 'Students'
Expression = {
$studentsText = $_.students
$studentsScriptBlock = Invoke-Expression -Command $studentsText
$studentsArray = #(& $studentsScriptBlock)
return $studentsArray
}
}
# Replace the [String] property "StudentNo" with an [Int32] property of the same name
$students = Import-Csv students.csv `
| Select-Object `
-ExcludeProperty 'StudentNo' `
-Property '*', #{
Name = 'StudentNo'
Expression = { [Int32] $_.StudentNo }
}
$combined = $students `
| Select-Object -Property `
'StudentNo', `
'PreferredSurname', `
'PreferredFirstnames', `
'UPN', `
#{
Name = "roomName";
Expression = {
foreach ($r in $rooms)
{
if ($r.Students.StudentNo -contains $_.StudentNo)
{
return $r.roomName
}
}
#TODO: Return text indicating room not found?
}
}
The reason this can be slow is because you are performing a linear search - two of them, in fact - for every student object; first through the collection of rooms (foreach), then through the collection of students in each room (-contains). This can quickly turn into a lot of iterations and equality comparisons because in every room to which the current student is not assigned you are iterating the entire collection of that room's students, on and on until you do find the room for that student.
One easy optimization you can make when performing a linear search is to sort the items you're searching (in this case, the Students property will be ordered by the StudentNo property of each student)...
# Replace the [String] property "students" with an array of [HashTable] property "Students"
$rooms = Import-Csv rooms.csv `
| Select-Object `
-ExcludeProperty 'students' `
-Property '*', #{
Name = 'Students'
Expression = {
$studentsText = $_.students
$studentsScriptBlock = Invoke-Expression -Command $studentsText
$studentsArray = #(& $studentsScriptBlock) `
| Sort-Object -Property #{ Expression = { $_.StudentNo } }
return $studentsArray
}
}
...and then when you're searching that same collection if you come across an item that is greater than the one you're searching for you know the remainder of the collection can't possibly contain what you're searching for and you can immediately abort the search...
#{
Name = "roomName";
Expression = {
foreach ($r in $rooms)
{
# Requires $room.Students to be sorted by StudentNo
foreach ($roomStudentNo in $r.Students.StudentNo)
{
if ($roomStudentNo -eq $_.StudentNo)
{
# Return the matched room name and stop searching this and further rooms
return $r.roomName
}
elseif ($roomStudentNo -gt $_.StudentNo)
{
# Stop searching this room
break
}
# $roomStudentNo is less than $_.StudentNo; keep searching this room
}
}
#TODO: Return text indicating room not found?
}
}
Better yet, with a sorted collection you can also perform a binary search, which is faster than a linear search*. The Array class already provides a BinarySearch static method, so we can accomplish this in less code, too...
#{
Name = "roomName";
Expression = {
foreach ($r in $rooms)
{
# Requires $room.Students to be sorted by StudentNo
if ([Array]::BinarySearch($r.Students.StudentNo, $_.StudentNo) -ge 0)
{
return $r.roomName
}
}
#TODO: Return text indicating room not found?
}
}
The way I would approach this problem, however, is to use a [HashTable] mapping a StudentNo to a room. There is a little preprocessing required to build the [HashTable] but this will provide constant-time lookups when retrieving the room for a student.
function GetRoomsByStudentNoTable()
{
$table = #{ }
foreach ($room in $rooms)
{
foreach ($student in $room.Students)
{
#NOTE: It is assumed each student belongs to at most one room
$table[$student.StudentNo] = $room
}
}
return $table
}
# Replace the [String] property "students" with an array of [HashTable] property "Students"
$rooms = Import-Csv rooms.csv `
| Select-Object `
-ExcludeProperty 'students' `
-Property '*', #{
Name = 'Students'
Expression = {
$studentsText = $_.students
$studentsScriptBlock = Invoke-Expression -Command $studentsText
$studentsArray = #(& $studentsScriptBlock)
return $studentsArray
}
}
# Replace the [String] property "StudentNo" with an [Int32] property of the same name
$students = Import-Csv students.csv `
| Select-Object `
-ExcludeProperty 'StudentNo' `
-Property '*', #{
Name = 'StudentNo'
Expression = { [Int32] $_.StudentNo }
}
$roomsByStudentNo = GetRoomsByStudentNoTable
$combined = $students `
| Select-Object -Property `
'StudentNo', `
'PreferredSurname', `
'PreferredFirstnames', `
'UPN', `
#{
Name = "roomName";
Expression = {
$room = $roomsByStudentNo[$_.StudentNo]
if ($room -ne $null)
{
return $room.roomName
}
#TODO: Return text indicating room not found?
}
}
You can ameliorate the hit of building $roomsByStudentNo by doing so at the same time as importing rooms.csv...
# Replace the [String] property "students" with an array of [HashTable] property "Students"
$rooms = Import-Csv rooms.csv `
| Select-Object `
-ExcludeProperty 'students' `
-Property '*', #{
Name = 'Students'
Expression = {
$studentsText = $_.students
$studentsScriptBlock = Invoke-Expression -Command $studentsText
$studentsArray = #(& $studentsScriptBlock)
return $studentsArray
}
} `
| ForEach-Object -Begin {
$roomsByStudentNo = #{ }
} -Process {
foreach ($student in $_.Students)
{
#NOTE: It is assumed each student belongs to at most one room
$roomsByStudentNo[$student.StudentNo] = $_
}
return $_
}
*Except for on small arrays

Change column title between columns in table of hash values

I have a table variable that can be treated like a csv file...
"SamAccountName","lastLogonTimestamp","AccountExpires","givenname","sn","distinguishedName","employeeNumber","employeeID","Description","extensionattribute8","userAccountControl"
"value1","value1","value1","value1","value1","value1","value1","value1","value1","value1","value1"
"value2","value2","value2","value2","value2","value2","value2","value2","value2","value2","value2"
"value3","value3","value3","value3","value3","value3","value3","value3","value3","value3","value3"
What I want to do, is change the two title names givenname to FirstName, and sn to LastName.
Note: I also want to change the values for lastLogonTimestamp and AccountExpires, but I already have the working code that does this. This code is as follows...
$listOfBadDateValues = '9223372036854775807', '9223372036854770000', '0'
$maxDateValue = '12/31/1600 5:00 PM'
$tableFixed = $table | % {
if ($_.lastLogonTimestamp) {
$_.lastLogonTimestamp = ([datetime]::FromFileTime($_.lastLogonTimestamp)).ToString('g')
}; if (($_.AccountExpires) -and ($listOfBadDateValues -contains $_.AccountExpires)) {
$_.AccountExpires = $null
} else {
if (([datetime]::FromFileTime($_.AccountExpires)).ToString('g') -eq $maxDateValue) {
$_.AccountExpires = $null
} Else {
$_.AccountExpires = ([datetime]::FromFileTime($_.AccountExpires)).ToString('g')
}
};$_}
How can I write the code so the two title names are changed to FirstName and LastName?
You can simply take the object and pipe it to a select statement to "alias" the property names.
$yourObject | Select-Object -Property #{N='MyNewName1';E={$_.ExistingName1}}, #{N='MyNewName2';E={$_.ExistingName2}}