PowerShell export multiple objects as csv - powershell

I have two columns of data the first is a string array the second is actually an object. I am looking for a simple way of exporting this as a csv. I have a version with a foreach loop that builds each string up but it seams like over kill. I have been trying to use select and select object to get it out somehow. Note I am just a beginner at powershell so I may be missing something.
My first attempt:
$data | Select-Object -ExpandProperty reports | Select -ExpandProperty data | Select -ExpandProperty rows | Format-Table $_.dimensions
Results in:
dimensions metrics
---------- -------
{New Visitor, "Mozilla} {#{values=System.Object[]}}
My second one went as far as looping
foreach ($report in $data.reports) {
"Rows:" + $report.data.rows.Count
foreach ($row in $report.data.rows) {
$output = ""
foreach($dim in $row.dimensions) {
$output += $dim + $seporator
}
foreach($met in $row.metrics) {
foreach($v in $met.values) {
$output += $v + $seporator
}
}
#| Out-File -Append C:\Users\linda_l\Documents\debug.txt
$output
}
}
There is a potential for a lot of data here so I would really like to avoid the string building solution.
Note: Data comes from the Google Analytics reporting api
$data = Invoke-RestMethod -ContentType 'application/json' -Uri "https://analyticsreporting.googleapis.com/v4/reports:batchGet?access_token=$($token.access_token)" -Method POST -Body $analyticsRequest
reports : {#{columnHeader=; data=}}
From comment:
$data | Export-Csv -Path "C:\Users\linda_l\Documents\debug2.csv"
System.Management.Automation.PSCustomObject reports System.Object[]
Optimal output csv
New Visitor,S40 Ovi Browser,1,2
New Visitor,Safari,3,4
Note its up on Github steps for getting a refreshtoken
Data is coming from the Google analytics reporting API

Related

web scraping using powershell

I am trying to scrape the pages of website https://www.enghindi.com/ .
URLs are saved in csv file, for example
URL
Hindi meaning
Url1
hindi meaning
url2
hindi meaning
now, everytime I am running following script . it just shows result of only URL1 and that goes into multiple cells. I want all result of url 1 should be in one cell (in hindi meaning box) and similarly for URL2.
url1 : https://www.enghindi.com/index.php?q=close
url2 : https://www.enghindi.com/index.php?q=compose
$URLs = import-csv -path C:\Scripts\PS\urls.csv | select -expandproperty urls
foreach ($url in $urls)
{
$web = Invoke-WebRequest $url
$data = $web.AllElements | Where{$_.TagName -eq "BIG"} | Select-Object -Expand InnerText
$datafinal = $data.where({$_ -like "*which*"},'until')
}
foreach ($item in $datafinal) {
[ pscustomobject]#{ Url = $url; Data = $item } | Export-Csv -Path C:\Scripts\PS\output.csv -NoTypeInformation -Encoding unicode -Append
}
Are there other ways I can get english to hindi word meaning using web scraping instead of copying and pasting. I prefer google translate but that I think difficult that is why i am trying with enghindi.com.
thanks alot
Web scraping, due its inherent unreliability, should only be a last resort.
You can make it work in Windows PowerShell, but note that the HTML DOM parsing is no longer available in PowerShell (Core) 7+.
You code has two basic problems:
It operates on $datafinal after the foreach loop, at which point you only see the results of the last Invoke-WebRequest call.
You loop over each element of array $datafinal and create an output object for each, instead of creating an output object per input URL.
The following reformulation fixes these problems:
# Sample input URLs
$URLs = #(
'https://www.enghindi.com/index.php?q=close',
'https://www.enghindi.com/index.php?q=compose'
)
$URLs |
ForEach-Object {
$web = Invoke-WebRequest $_
$data = $web.AllElements | Where { $_.TagName -eq "BIG" } | Select-Object -Expand InnerText
$datafinal = $data.where({ $_ -like "*which*" }, 'until')
# Create the output object for the URL at hand and implicitly output it.
# Join the $datafinal elements with newlines to form a single vaulue.
[pscustomobject] #{
Url = $_
Hindi = $datafinal -join "`n"
}
} |
ConvertTo-Csv -NoTypeInformation
Note that, for demonstration purposes, ConvertTo-Csv is used in lieu of Export-Csv, which allows you to see the results instantly.

Running ForEach-Object -Parallel, data missing from export

I have some working code that basically queries 2 different Graph API endpoints, then searches for a match in the User Principal Name column, and inserts the extension_335d4df9847945fbaa472c8b8fbb5d75_employeeNumber column and values to the exported csv (Thanks to the user #PMental for this solution) This column derives from attribute that was recently extended from our on premises AD.
This code works perfectly fine, however if I try to parallelize it, I get no results in the extension_335d4df9847945fbaa472c8b8fbb5d75_employeeNumber column.
Is this because once it is being parallelized, I'm not able to share variables between the parallel processes? If so, how on earth do I accomplish this?
Code below - if you remove the -Parallel, it works fine:
$graphApiUri = "https://graph.microsoft.com/v1.0/reports/getOffice365ActiveUserDetail(period='D90')"
$Uri = "https://graph.microsoft.com/v1.0/users?`$select=userPrincipalName,extension_335d4df9847945fbaa472c8b8fbb5d75_employeeNumber"
$O365Report = Invoke-RestMethod -Method Get -Uri $graphApiUri -Headers $headerParams | ConvertFrom-Csv
# If the result is more than 999, we need to read the #odata.nextLink to show more than one side of users
$UserDetails = while (-not [string]::IsNullOrEmpty($uri)) {
# API Call
$apiCall = try {
Invoke-RestMethod -Headers $headerParams -Uri $uri -Method Get
}
catch {
$errorMessage = $_.ErrorDetails.Message | ConvertFrom-Json
}
$uri = $null
if ($apiCall) {
# Check if any data is left
$uri = $apiCall.'#odata.nextLink'
$apiCall
}
}
Write-Output "Matching UPN to employeeNumber..."
$O365Report | ForEach-Object -Parallel {
$CurrentEmpNumber = $UserDetails.value |
Where-Object userPrincipalName -eq $_.'User Principal Name' |
Select-Object -ExpandProperty extension_335d4df9847945fbaa472c8b8fbb5d75_employeeNumber -ErrorAction SilentlyContinue
$_ | Add-Member -MemberType NoteProperty -Name extension_335d4df9847945fbaa472c8b8fbb5d75_employeeNumber -Value $CurrentEmpNumber
}
$O365Report | Export-Csv $ReportCSV -NoTypeInformation
Write-Output "Report saved to $ReportCSV."
When inside of a ForEach-Object -Parallel script block, and you are trying to reference variables which were created outside of it, you need to preface the variable name with using: so it would be $using:UserDetails
Examples:
Returns nothing because $test isn't accessible within the scope of the parallel script block:
$test = 1;
0..5 | % -Parallel { $test; };
Returns the value of $test five times because by using $using:test you are now able to see its value:
$test = 1;
0..5 | % -Parallel { $using:test; };
From documenation:
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object?view=powershell-7.1
The ForEach-Object -Parallel parameter set runs script blocks in parallel on separate process threads. The $using: keyword allows passing variable references from the cmdlet invocation thread to each running script block thread. Since the script blocks run in different threads, the object variables passed by reference must be used safely. Generally it is safe to read from referenced objects that don't change. But if the object state is being modified then you must used thread safe objects, such as .Net System.Collection.Concurrent types (See Example 11).
Personal note:
I would also recommend using -ThrottleLimit to limit its max degrees of paralellism. The default is 5, but you may want more or less than that depending on testing.

How do I get the data results from Invoke-RestMethod into CSV file?

I'm trying to get the results (data) from a REST call into a CSV file.
My code is:
$results = Invoke-RestMethod -Method Get -Uri $url -Header $bearer -ContentType 'application/json'
$results
and it comes back with:
data
----
{#{type=flight-offer; id=1559566119876--1838046263; offerItems=System.Object[]}, #{type=flight-offer; id=15595...
I want to get the values from each offerItems object into a CSV file.
Expand the data property and export the nested objects to a CSV:
$results |
Select-Object -Expand data |
Export-Csv 'C:\path\to\output.csv' -NoType
However, since your data objects apparently contain an array (offerItems=System.Object[]) you'll have to mangle that array into a string first, e.g. like this:
... |
Select-Object -Expand data |
Select-Object *,#{n='offerItems';e={$_.offerItems -join ','}} -Exclude offerItems |
...
The merge operation may differ depending on what kind of data the array holds.

Output CSV using Powershell

I got a problem with understanding of output CSV file from powershell script!
I have a Request to restapi server and I'm getting a variable which contain 10 or more lines, like:
>$ExpenseDescription
Taxi to airport
Taxi to seaport
Taxi to spaceport
Taxi to home
And the I'm creating table
$tableExpenses=#"
Created On | Description | Expense Category
$ExpenseCreatedOnt | $ExpenseDescription |$ExpenseReport
$tableExpense|Out-File C:\Users\book.xls
$tableExpense|Out-File C:\Users\book.csv
And as output file I'm getting .xls and .csv!
So the problem is that I have 10 lines in variable $ExpenseDescriptionand the OutFile contain all 10 lines in 1 cell in book.xls!
How can I split them in code and make OutFile in format like this:
Created On | Description | Expense Category
10.10.2018|Taxi to airport| Money
11.10.2018|Taxi to seaport| Visa
Because now I'm having this in output
Created On | Description | Expense Category
10.10.2018 11.10.2018|Taxi to airport Taxi to seaport| Money Visa|
OK, I'll add more code)
WebRequest
$ReportURI = ("https://api.rest.com/data/query")
$ReportQuery =
#{"q"="SELECT Category,Description,CreatedOn from Expense"}
Try
{$ResponseReport = Invoke-RestMethod -Method Post -Uri $ReportURI -Headers #{"Authorization" = $SessionId} -Body ( $ReportQuery | ConvertTo-Json) -ContentType "application/json" -ErrorAction Stop}
Write-Host $ResponseReport}
ConvertTo-Json $ResponseReport
variables
$ExpenseCreatedOn = $ResponseReport.CreatedOn
$ExpenseDescription = $ResponseReport.Description
$ExpenseReport = $ResponseReport.Category.Name
table_format
$tableExpense=#"
Created On Description Expense Category
$ExpenseCreatedOn $ExpenseDescription $ExpenseReport
$tableExpense|Out-File C:\Users\book.xls
$tableExpense|Out-File C:\Users\book.csv
You're not outputting a CSV. With Out-File, you're exporting a text file.
Providing that your variables hold an array of strings, you could index into them to create an object, then use Export-Csv to export that:
foreach($i in 0..($ExpenseDescription.Count - 1)){
[array]$tableExpenses += [pscustomobject]#{
"Created On" = $ExpenseCreatedOnt[$i]
Description = $ExpenseDescription[$i]
"Expense Category" = $ExpenseReport[$i]
}
}
$tableExpenses | Export-Csv C:\Users\book.csv -NoType
$tableExpenses | Export-Csv C:\Users\book2.csv -NoType -Delimiter "|"

PowerShell TFS REST-API object loop advise

I have a piece of code that i managed to get working, but i feel that it can be written a lot easier. Im new with PowerShell and am trying to understand it better. I have a double foreach below to get the key and value out of the PSCustomObject that comes out of the TFS REST-API call.
For some reason im doing 2 loops, but i dont understand why this is required.
A sample of the contents of $nameCap.userCapabilities is
Name1 Name2
----- -----
Value1 Value2
So basically i want to loop over the "name/value pairs" and get their values.
What can i do better ?
$uri = "$tfsUri/_apis/distributedtask/pools/$global:agentPoolId/agents?api-version=3.0-preview&includeCapabilities=true"
$result = (Invoke-RestMethod -Uri $uri -Method Get -ContentType "application/json" -UseDefaultCredentials).value | select name, userCapabilities, systemCapabilities
#Loop over all agents and their capablities
foreach ($nameCap in $result)
{
$capabilityNamesList = New-Object System.Collections.ArrayList
#Loop over all userCapabilities and store their names
#($nameCap.userCapabilities) | %{
$current_Cap = $_
$req_cap_exists = $false
Get-Member -MemberType Properties -InputObject $current_Cap | %{
$temp_NAME = $_.Name
$temp_Value = Select-Object -InputObject $current_Cap -ExpandProperty $_.Name
[void]$capabilityNamesList.Add($temp_NAME)
}
}
}
I mean if you just need the Name and value, like userCapabilities, then just select for it.
so:
$result | select Name,userCapabilites
And if it doesn't give you a table automatically, then | ft -force