Powershell pscustomobject format-table new row instead of one line - powershell

I have a very large JSON response for employees that I am trying to get into table format, export to CSV and eventually insert into SQL Server. I was able to determine how to get all of my variables from the json file, however now I am getting all of my values inserted on one row for each column instead of a new row for each employee.
Also, when I export to CSV the value turns into System.Object[].
$json1 = Invoke-webRequest -Uri $Workeruri -Certificate $cert -Headers $WorkerHeader | convertfrom-json
$table = [PSCustomObject] #{
associateOID = $json1.workers.associateOID
workerID = $json1.workers.workerID.idValue
GivenName = $json1.workers.person.legalName.givenName
MiddleName = $json1.workers.person.legalName.middleName
FamilyName1 = $json.workers.person.legalName.familyName1
} |format-table -autosize
$table | export-csv $filepath -NoTypeInformation
The columns are a small sample, there are actually probably 100 columns. However, my response returns like this:
associateOID workerID givenName
------------ -------- ---------
{1,2,3,4,5...} {a,b,c,d,e...} {Lebron James, Micheal Jordan, Steph Curry...}
I would like it to return:
associateOID workerID givenName
------------ -------- ---------
1 A Lebron James
2 B Micheal Jordan
3 C Steph Curry
Also, when exporting to CSV the response has the correct columns, but all columns return with: System.Object[].
Also, my fields that have ints and dates are not returning data. How can I fix that as well?
I have tried using sort-object, group-object, for-each loops. Nothing has worked.

you can try like this:
$json1 = Invoke-webRequest -Uri $Workeruri -Certificate $cert -Headers $WorkerHeader | ConvertFrom-Json
$table = $json1 | ForEach-Object {
[PSCustomObject] #{
associateOID = $_.workers.associateOID
workerID = $_.workers.workerID.idValue
GivenName = $_.workers.person.legalName.givenName
MiddleName = $_.workers.person.legalName.middleName
FamilyName1 = $_.workers.person.legalName.familyName1
}
}
$table | Export-Csv $filepath -NoTypeInformation
$table | Format-Table -AutoSize
Your snippet takes all the values for each column and stores them in a single object instead of iterating on the object collection converted from JSON.
Also, once you use Format-Table, data is formatted for display but not usable in the pipeline anymore. That's why I've separated display on screen and CSV export.

#sodawillow almost had it, assuming that json1.workers is the list of objects that contains your workers.
$json1 = Invoke-webRequest -Uri $Workeruri -Certificate $cert -Headers $WorkerHeader | ConvertFrom-Json
$table = $json1.workers | ForEach-Object {
[PSCustomObject] #{
associateOID = $_.associateOID
workerID = $_.workerID.idValue
GivenName = $_.person.legalName.givenName
MiddleName = $_.person.legalName.middleName
FamilyName1 = $_.person.legalName.familyName1
}
}
$table | Export-Csv $filepath -NoTypeInformation
$table | Format-Table -AutoSize

Related

Powershell - Using ConvertFrom-csv

I'm brand new to Powershell. I have a variable that contains comma separated values. What I want to do is read each entry in the csv string variable, and assign it to a variable. I am using ConvertFrom-csv to separate the data with headers.
How can I assign each value to a variable, or even better, use ConvertTo-csv to create a new csv string which only has, for example, columns 2/3/6/7 in it?
I would ultimately want to write that data out to a new csv file.
Here is my test code:
#Setup the variable
$Data = "test1,test2,test3,1234,5678,1/1/2021,12/31/2021"
$Data | ConvertFrom-csv -Header Header1,Header2, Header3, Header4, Header5, Header6, Header7
# Verify that an object has been created.
$Data |
ConvertFrom-csv -Header Header1,Header2, Header3, Header4, Header5, Header6, Header7 |
Get-Member
#Show header1
Write-Host "--------Value from $Data----------------------------------------"
$Data[0] #doesn't work, only displays the first character of the string
Write-Host "-----------------------------------------------------------------"
Let me suggest a different approach. If you use ConvertFrom-Csv and assign the result of a variable ($data), this will be an array of Custom Objects. You can run this through a loop that steps through the elements of the array , one at a time, and then through an inner loop that steps through the properties of each object one at a time, setting a variable with the same name as the field header and the same value as the current record's value.
I don't have code that does exactly what you want. But I'm including code that I wrote a few years back that does something similar only using Import-Csv instead of ConverFrom-Csv.
Import-Csv $driver | % {
$_.psobject.properties | % {Set-variable -name $_.name -value $_.value}
Get-Content $template | % {$ExecutionContext.InvokeCommand.ExpandString($_)}
}
Focus on the first inner loop. Each property of the current object will have a name that came from the header and a value that came from the current record of the Csv file. You can ignore the line that says ExpandString. That's just what I choose to do with the variables once they have been defined.
How can I assign each value to a variable, or even better, use ConvertTo-Csv to create a new csv string which only has, for example, columns 2/3/6/7 in it?
This is one way of automating this:
# Define the CSV without headers
$Data = "test1,test2,test3,1234,5678,1/1/2021,12/31/2021"
# Set the number of headers needed
$headers = $Data.Split(',') | ForEach-Object -Begin { $i = 1 } -Process {
"Header$i"; $i++
}
# Set the desired columns we want
$desiredColumns = 2,3,6,7 | ForEach-Object { $_ - 1 } | ForEach-Object {
$headers[$_]
}
# Convert to CSV and filter by Desired Columns
$Data | ConvertFrom-Csv -Header $headers | Select-Object $desiredColumns
Result
Header2 Header3 Header6 Header7
------- ------- ------- -------
test2 test3 1/1/2021 12/31/2021
Result as CSV
$Data | ConvertFrom-Csv -Header $headers |
Select-Object $desiredColumns | ConvertTo-Csv -NoTypeInformation
"Header2","Header3","Header6","Header7"
"test2","test3","1/1/2021","12/31/2021"

Powershell Compare-object IF different then ONLY list items from one file, not both

I have deleted my original question because I believe I have a more efficient way to run my script, thus I'm changing my question.
$scrubFileOneDelim = "|"
$scrubFileTwoDelim = "|"
$scrubFileOneBal = 2
$scrubFileTwoBal = 56
$scrubFileOneAcctNum = 0
$scrubFileTwoAcctNum = 0
$ColumnsF1 = Get-Content $scrubFileOne | ForEach-Object{($_.split($scrubFileOneDelim)).Count} | Measure-Object -Maximum | Select-Object -ExpandProperty Maximum
$ColumnsF2 = Get-Content $scrubFileTwo | ForEach-Object{($_.split($scrubFileTwoDelim)).Count} | Measure-Object -Maximum | Select-Object -ExpandProperty Maximum
$useColumnsF1 = $ColumnsF1-1;
$useColumnsF2 = $ColumnsF2-1;
$fileOne = import-csv "$scrubFileOne" -Delimiter "$scrubFileOneDelim" -Header (0..$useColumnsF1) | select -Property #{label="BALANCE";expression={$($_.$scrubFileOneBal)}},#{label="ACCTNUM";expression={$($_.$scrubFileOneAcctNum)}}
$fileTwo = import-csv "$scrubFileTwo" -Delimiter "$scrubFileTwoDelim" -Header (0..$useColumnsF2) | select -Property #{label="BALANCE";expression={$($_.$scrubFileTwoBal)}},#{label="ACCTNUM";expression={$($_.$scrubFileTwoAcctNum)}}
$hash = #{}
$hashTwo = #{}
$fileOne | foreach { $hash.add($_.ACCTNUM, $_.BALANCE) }
$fileTwo | foreach { $hashTwo.add($_.ACCTNUM, $_.BALANCE) }
In this script I'm doing the following, counting header's to return the count and use it in a range operator in order to dynamically insert headers for later manipulation. Then I'm importing 2 CSV files. I'm taking those CSV files and pushing them into their own hashtable.
Just for an idea of what I'm trying to do from here...
CSV1 (as a hashtable) looks like this:
Name Value
---- -----
000000000001 000000285+
000000000002 000031000+
000000000003 000004685+
000000000004 000025877+
000000000005 000000001+
000000000006 000031000+
000000000007 000018137+
000000000008 000000000+
CSV2 (as a hashtable) looks like this:
Name Value
---- -----
000000000001 000008411+
000000000003 000018137+
000000000007 000042865+
000000000008 000009761+
I would like to create a third hash table. It will have all the "NAME" items from CSV2, but I don't want the "VALUE" from CSV2, I want it to have the "VALUE"s that CSV1 has. So in the end result would look like this.
Name Value
---- -----
000000000001 000000285+
000000000003 000004685+
000000000007 000018137+
000000000008 000000000+
Ultimately I want this to be exported as a csv.
I have tried this with just doing a compare-object, not doing the hashtables with the following code, but I abandoned trying to do it this way because file 1 may have 100,000 "accounts" where file 2 only has 200, and the result I was getting listed close to the 100,000 accounts that I didn't want to be in the result. They had the right balances but I want a file that only has those balances for the accounts listed in file 2. This code below isn't really a part of my question, just showing something I've tried. I just think this is much easier and faster with a hash table now so I would like to go that route.
#Find and Rename the BALANCE and ACCOUNT NUMBER columns in both files.
$fileOne = import-csv "$scrubFileOne" -Delimiter "$scrubFileOneDelim" -Header (0..$useColumnsF1) | select -Property #{label="BALANCE";expression={$($_.$scrubFileOneBal)}},#{label="ACCT-NUM";expression={$($_.$scrubFileOneAcctNum)}}
$fileTwo = import-csv "$scrubFileTwo" -Delimiter "$scrubFileTwoDelim" -Header (0..$useColumnsF2) | select -Property #{label="BALANCE";expression={$($_.$scrubFileTwoBal)}},#{label="ACCT-NUM";expression={$($_.$scrubFileTwoAcctNum)}}
Compare-Object $fileOne $fileTwo -Property 'BALANCE','ACCTNUM' -IncludeEqual -PassThru | Where-Object{$_.sideIndicator -eq "<="} | select * -Exclude SideIndicator | export-csv -notype "C:\test\f1.txt"
What you are after is filtering the Compare-Object function. This will show only one side of the result. YOu will need to place this before you exclude that property for it to work.
| Where-Object{$_.sideIndicator -eq "<="} |
Assuming that you have the following hash tables:
$hash = #{
'000000000001' = '000000285+';
'000000000002' = '000031000+';
'000000000003' = '000004685+';
'000000000004' = '000025877+';
'000000000005' = '000000001+';
'000000000006' = '000031000+';
'000000000007' = '000018137+';
'000000000008' = '000000000+';
}
$hashTwo = #{
'000000000001' = '000008411+';
'000000000003' = '000018137+';
'000000000007' = '000042865+';
'000000000008' = '000009761+';
}
you can create the third hash table by iterating over the keys from the second hash table and then assigning those keys to the value from the first hash table.
$hashThree = #{}
ForEach ($key In $hashTwo.Keys) {
$hashThree["$key"] = $hash["$key"]
}
$hashThree
The output of $hashThree is:
Name Value
---- -----
000000000007 000018137+
000000000001 000000285+
000000000008 000000000+
000000000003 000004685+
If you want the order of the data maintained (and you are using PowerShell 6 Core), you can use [ordered]#{} when creating the hash tables.

Output CSV using Powershell

I got a problem with understanding of output CSV file from powershell script!
I have a Request to restapi server and I'm getting a variable which contain 10 or more lines, like:
>$ExpenseDescription
Taxi to airport
Taxi to seaport
Taxi to spaceport
Taxi to home
And the I'm creating table
$tableExpenses=#"
Created On | Description | Expense Category
$ExpenseCreatedOnt | $ExpenseDescription |$ExpenseReport
$tableExpense|Out-File C:\Users\book.xls
$tableExpense|Out-File C:\Users\book.csv
And as output file I'm getting .xls and .csv!
So the problem is that I have 10 lines in variable $ExpenseDescriptionand the OutFile contain all 10 lines in 1 cell in book.xls!
How can I split them in code and make OutFile in format like this:
Created On | Description | Expense Category
10.10.2018|Taxi to airport| Money
11.10.2018|Taxi to seaport| Visa
Because now I'm having this in output
Created On | Description | Expense Category
10.10.2018 11.10.2018|Taxi to airport Taxi to seaport| Money Visa|
OK, I'll add more code)
WebRequest
$ReportURI = ("https://api.rest.com/data/query")
$ReportQuery =
#{"q"="SELECT Category,Description,CreatedOn from Expense"}
Try
{$ResponseReport = Invoke-RestMethod -Method Post -Uri $ReportURI -Headers #{"Authorization" = $SessionId} -Body ( $ReportQuery | ConvertTo-Json) -ContentType "application/json" -ErrorAction Stop}
Write-Host $ResponseReport}
ConvertTo-Json $ResponseReport
variables
$ExpenseCreatedOn = $ResponseReport.CreatedOn
$ExpenseDescription = $ResponseReport.Description
$ExpenseReport = $ResponseReport.Category.Name
table_format
$tableExpense=#"
Created On Description Expense Category
$ExpenseCreatedOn $ExpenseDescription $ExpenseReport
$tableExpense|Out-File C:\Users\book.xls
$tableExpense|Out-File C:\Users\book.csv
You're not outputting a CSV. With Out-File, you're exporting a text file.
Providing that your variables hold an array of strings, you could index into them to create an object, then use Export-Csv to export that:
foreach($i in 0..($ExpenseDescription.Count - 1)){
[array]$tableExpenses += [pscustomobject]#{
"Created On" = $ExpenseCreatedOnt[$i]
Description = $ExpenseDescription[$i]
"Expense Category" = $ExpenseReport[$i]
}
}
$tableExpenses | Export-Csv C:\Users\book.csv -NoType
$tableExpenses | Export-Csv C:\Users\book2.csv -NoType -Delimiter "|"

PowerShell TFS REST-API object loop advise

I have a piece of code that i managed to get working, but i feel that it can be written a lot easier. Im new with PowerShell and am trying to understand it better. I have a double foreach below to get the key and value out of the PSCustomObject that comes out of the TFS REST-API call.
For some reason im doing 2 loops, but i dont understand why this is required.
A sample of the contents of $nameCap.userCapabilities is
Name1 Name2
----- -----
Value1 Value2
So basically i want to loop over the "name/value pairs" and get their values.
What can i do better ?
$uri = "$tfsUri/_apis/distributedtask/pools/$global:agentPoolId/agents?api-version=3.0-preview&includeCapabilities=true"
$result = (Invoke-RestMethod -Uri $uri -Method Get -ContentType "application/json" -UseDefaultCredentials).value | select name, userCapabilities, systemCapabilities
#Loop over all agents and their capablities
foreach ($nameCap in $result)
{
$capabilityNamesList = New-Object System.Collections.ArrayList
#Loop over all userCapabilities and store their names
#($nameCap.userCapabilities) | %{
$current_Cap = $_
$req_cap_exists = $false
Get-Member -MemberType Properties -InputObject $current_Cap | %{
$temp_NAME = $_.Name
$temp_Value = Select-Object -InputObject $current_Cap -ExpandProperty $_.Name
[void]$capabilityNamesList.Add($temp_NAME)
}
}
}
I mean if you just need the Name and value, like userCapabilities, then just select for it.
so:
$result | select Name,userCapabilites
And if it doesn't give you a table automatically, then | ft -force

Compare-Object Output Format

I have two CSV files I need to compare. Both of which may (or may not) contain an additional delimited field (delimited via a "|" character), like so:
(new.csv)
Title,Key,Value
Jason,Son,Hair=Red|Eyes=Blue
James,Son,Hair=Brown|Eyes=Green
Ron,Father,Hair=Black
Susan,Mother,Hair=Black|Eyes=Brown|Dress=Green
(old.csv)
Title,Key,Value
Jason,Son,Hair=Purple|Eyes=Blue
James,Son,Hair=Brown|Eyes=Green
Ron,Father,Hair=Purple
Susan,Mother,Hair=Black|Eyes=Brown|Dress=Blue
My problem comes in when I attempt to compare the two files...
$fileNew = "new.csv"
$fileOld = "old.csv"
$fileDiffOutputFile = "diff.txt"
$csvNewLog = (Import-CSV ($fileNew))
$csvOldLog = (Import-CSV ($fileOld))
$varDifferences = Compare-Object $csvOldLog $csvNewLog -property Title,Value
$varDifferences | Group-Object -Property Title | % {New-Object -TypeName PSObject -Property #{ NewValue=($_.group[0].Value); Title=$_.name; OldValue=($_.group[1].Value) } } | Out-File $fileDiffOutputFile -Append
Resulting in this output:
(diff.txt)
OldValue Title NewValue
-------- ----- --------
Hair=Purple|Eyes=Blue Jason Hair=Red|Eyes=Blue
Hair=Purple Ron Hair=Black
Hair=Black|Eyes=Brown|D... Susan Hair=Black|Eyes=Brown|...
Some of the values are inevitably going to extend out past the max length of the column, as it does with Susan above.
So, my question could have a couple of solutions that I can think of:
Is there an easier way to isolate the values so that I only pull out the changed values, and not the entire string of delimited values?
If not, is it possible to get the format to show the entire string (including the unchanged values part of the delimited string) instead?
If you include a format-table -wrap in your last line, like so?
$fileNew = "new.csv"
$fileOld = "old.csv"
$fileDiffOutputFile = "diff.txt"
$csvNewLog = (Import-CSV ($fileNew))
$csvOldLog = (Import-CSV ($fileOld))
$varDifferences = Compare-Object $csvOldLog $csvNewLog -property Title,Value
$varDifferences | Group-Object -Property Title | % {New-Object -TypeName PSObject -Property #{ NewValue=($_.group[0].Value); Title=$_.name; OldValue=($_.group[1].Value) } } | Format-Table -wrap | Out-File $fileDiffOutputFile -Append