add element to array in powershell - powershell

I'm trying to add a Totals line to the array $response_table.
The line should be Totals = 56 (number), for example.
I've tried $response_table += #{Name="Total"; Count=55};, it adds a line with rubbish data.
Can you help with how to add this?
Code is below
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12
$current_month = (Get-Date).month;
$current_year = (Get-Date).year;
$response_table=#();
$what_year=2019
$month_count=12
$current_month, $total_year, $current_date_interval;
$total_monthly=#(); # empty array, will be populated with values from the API
#$numbers += 5, 2, 3; # to add values to array
if ($what_year -eq $current_year)
{
$month_count = $current_month-1;
}
for ($current_month=1; $current_month -le $month_count; $current_month++)
{
$current_date_interval = -join($what_year, "-", $current_month);
$uri="https://data.police.uk/api/crimes-street/bicycle-theft?poly=53.950624,-1.059234:53.951301,-1.049181:53.947361,-1.043420:53.950333,-1.030671:53.952997,-1.016427:53.950189,-1.013653:53.929487,-1.042286:53.942512,-1.054948:53.941936,-1.057172:53.944481,-1.060155s8&date="+$current_date_interval;
$response = Invoke-RestMethod -Uri $uri -Method Get -Headers $headers
$response_table += $response | Group month |Select -Property name,count
Write-Host $response_table;
$response | Group month |Select -Property name,count
$total_year += $response.count;
}
Write-Host ($response_table|Measure-object -Property count -sum)
$response_table += #{Name="Total"; Count=55};
# does not work
$response_table | export-csv "list.csv" -notypeinformation
#add-content "list.csv" "Total,$total_year"
Write-Host "Yearly total"$total_year;

As AdminOfThings indicates, you're mistakenly adding a hashtable to the existing array of custom objects (that Select-Object outputs).
To construct a custom object in PSv3+, simply place [pscustomobject] before a hashtable literal, which in your case means:
$response_table += [pscustomobject] #{Name="Total"; Count=55}
Now your Export-Csv command should work as intended.
As an aside: Up to PowerShell 6.x, Export-Csv only (meaningfully) supports custom objects as input, not also hashtables. v7 adds support for hashtables.
Generally avoid using += for appending to arrays:
Arrays are fixed-size data structures, so what PowerShell must do when you "append to" an array with += is to create a new array behind the scenes every time, which is quite inefficient in a loop.
While using an efficiently in-place extensible data structure such as [System.Collections.ArrayList] or [System.Collections.Generic.List[object]] is the next best thing, the simplest and fastest approach is to simply let PowerShell collect multiple output objects in an array ([object[]]) for you, by assigning the entire loop as an expression to a variable:
[array] $response_table = for ($current_month=1; $current_month -le $month_count; $current_month++)
{
$current_date_interval = -join($what_year, "-", $current_month);
$uri="https://data.police.uk/api/crimes-street/bicycle-theft?poly=53.950624,-1.059234:53.951301,-1.049181:53.947361,-1.043420:53.950333,-1.030671:53.952997,-1.016427:53.950189,-1.013653:53.929487,-1.042286:53.942512,-1.054948:53.941936,-1.057172:53.944481,-1.060155s8&date="+$current_date_interval;
$response = Invoke-RestMethod -Uri $uri -Method Get -Headers $headers
# *Output* the result of this pipeline, and PowerShell will
# collect all outputs for you
$response | Group month | Select -Property name,count
$total_year += $response.count;
}
# It's fine to use += to add the total, because you're only using it *once*.
$response_table += [pscustomobject] #{Name="Total"; Count=55}
Note: The [array] type constraint ensures that $response_table becomes an array even if the loop happens to have just 1 iteration.

Related

Running ForEach-Object -Parallel, data missing from export

I have some working code that basically queries 2 different Graph API endpoints, then searches for a match in the User Principal Name column, and inserts the extension_335d4df9847945fbaa472c8b8fbb5d75_employeeNumber column and values to the exported csv (Thanks to the user #PMental for this solution) This column derives from attribute that was recently extended from our on premises AD.
This code works perfectly fine, however if I try to parallelize it, I get no results in the extension_335d4df9847945fbaa472c8b8fbb5d75_employeeNumber column.
Is this because once it is being parallelized, I'm not able to share variables between the parallel processes? If so, how on earth do I accomplish this?
Code below - if you remove the -Parallel, it works fine:
$graphApiUri = "https://graph.microsoft.com/v1.0/reports/getOffice365ActiveUserDetail(period='D90')"
$Uri = "https://graph.microsoft.com/v1.0/users?`$select=userPrincipalName,extension_335d4df9847945fbaa472c8b8fbb5d75_employeeNumber"
$O365Report = Invoke-RestMethod -Method Get -Uri $graphApiUri -Headers $headerParams | ConvertFrom-Csv
# If the result is more than 999, we need to read the #odata.nextLink to show more than one side of users
$UserDetails = while (-not [string]::IsNullOrEmpty($uri)) {
# API Call
$apiCall = try {
Invoke-RestMethod -Headers $headerParams -Uri $uri -Method Get
}
catch {
$errorMessage = $_.ErrorDetails.Message | ConvertFrom-Json
}
$uri = $null
if ($apiCall) {
# Check if any data is left
$uri = $apiCall.'#odata.nextLink'
$apiCall
}
}
Write-Output "Matching UPN to employeeNumber..."
$O365Report | ForEach-Object -Parallel {
$CurrentEmpNumber = $UserDetails.value |
Where-Object userPrincipalName -eq $_.'User Principal Name' |
Select-Object -ExpandProperty extension_335d4df9847945fbaa472c8b8fbb5d75_employeeNumber -ErrorAction SilentlyContinue
$_ | Add-Member -MemberType NoteProperty -Name extension_335d4df9847945fbaa472c8b8fbb5d75_employeeNumber -Value $CurrentEmpNumber
}
$O365Report | Export-Csv $ReportCSV -NoTypeInformation
Write-Output "Report saved to $ReportCSV."
When inside of a ForEach-Object -Parallel script block, and you are trying to reference variables which were created outside of it, you need to preface the variable name with using: so it would be $using:UserDetails
Examples:
Returns nothing because $test isn't accessible within the scope of the parallel script block:
$test = 1;
0..5 | % -Parallel { $test; };
Returns the value of $test five times because by using $using:test you are now able to see its value:
$test = 1;
0..5 | % -Parallel { $using:test; };
From documenation:
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object?view=powershell-7.1
The ForEach-Object -Parallel parameter set runs script blocks in parallel on separate process threads. The $using: keyword allows passing variable references from the cmdlet invocation thread to each running script block thread. Since the script blocks run in different threads, the object variables passed by reference must be used safely. Generally it is safe to read from referenced objects that don't change. But if the object state is being modified then you must used thread safe objects, such as .Net System.Collection.Concurrent types (See Example 11).
Personal note:
I would also recommend using -ThrottleLimit to limit its max degrees of paralellism. The default is 5, but you may want more or less than that depending on testing.

`Invoke-RestMethod -Uri "..." -Method Get | select X,Y` doesn't return any row while `(Invoke-RestMethod -Uri "..." -Method Get) | select X,Y` does?

I have a rest api which will return rows. However,
Why Invoke-RestMethod -Uri "..." -Method Get | select XXX,YYY returns just header?
XXX YYY
--- ---
but (Invoke-RestMethod -Uri "..." -Method Get) | select X,Y returns rows?
Assigning to a variable first also works.
$x = Invoke-RestMethod -Uri "..." -Method Get | select XXX,YYY
$x | select xxx,yyyy
Generally speaking:
If a command outputs collections as single objects, Select-Object X, Y won't work as expected, because it'll look for those properties on the collection object, where they can't be found, in which case Select-Object creates a single object with the requested properties, all of which then contain $null.
Invoke-RestMethod is a likely candidate for this behavior, because it may implicitly parse the return value as JSON via ConvertFrom-Json, which indeed outputs arrays as single objects; this surprising behavior is discussed in this GitHub issue.
Placing (...) around a command forces enumeration, so that may solve the problem:
# Place (...) around the Invoke-RestMethod call to force enumeration.
(Invoke-RestMethod -Uri "..." -Method Get) | select XXX,YY
Another option is to assign to an (intermediate) variable, as shown in your question - though the (...) approach is simpler, if you don't actually need to store the intermediate result.
# Store array in a variable.
$array = Invoke-RestMethod -Uri "..." -Method Get
# An array stored in a variable sent through the pipeline is
# invariably enumerated.
$array | select X,Y
This works, because sending an array stored in a variable through the pipeline always enumerates it (sends it elements one by one).
By assigning to a variable you effectively obliterate the distinction between a command that outputs N objects one by one and one that outputs an N-element array as a single object:
# Send an array *as a whole* through the pipeline.
PS> (Write-Output -NoEnumerate (1..3) | Measure-Object).Count
1 # That is, the 3-element array was sent as *one* object
# Wrapping the command in (...) forces enumeration.
PS> ((Write-Output -NoEnumerate (1..3)) | Measure-Object).Count
3 # elements were sent *one by one*
# Store output-as-a-whole array in a variable,
# then send the variable through the pipeline -
# which also forces enumeration.
PS> $array = Write-Output -NoEnumerate (1..3); ($array | Measure-Object).Count
3 # elements were sent *one by one*
Conversely, if you do want to send an array stored in a variable as a whole through the pipeline, you have two options:
$array = 1..3
# Use Write-Output -NoEnumerate:
PS> (Write-Output -NoEnumerate $array | Measure-Object).Count
1 # array was sent *as a whole*
# Alternatively - faster and more concise, but more obscure -
# wrap the array in an aux. wrapper array, so that only
# the wrapper array is enumerated, sending the original array
# as a whole:
PS> (, $array | Measure-Object).Count
1 # array was sent *as a whole*

Powershell Foreach not skipping values from array

I am trying to write a script that downloads web sites information. I am able to download the information but I cannot seem to get the filtering working. I have an a series of values that I want skipped stored in $TakeOut but it does not recognize the values in the if -eq $TakeOut. I have to write a line for each value.
What I am wondering is, if there is a way to use a $value as over time there will be a considerable amount of values to skip.
This works but is not practical in the long run.
if ($R.innerText -eq "Home") {Continue}
Something like this would be preferable.
if ($R.innerText -eq $TakeOut) {Continue}
Here is a sample of my code.
#List of values to skip
$TakeOut = #()
$TakeOut = (
"Help",
"Home",
"News",
"Sports",
"Terms of use",
"Travel",
"Video",
"Weather"
)
#Retrieve website information
$Results = ((Invoke-WebRequest -Uri "https://www.msn.com/en-ca/").Links)
#Filter and format to new table of values
$objects = #()
foreach($R in $Results) {
if ($R.innerText -eq $TakeOut) {Continue}
$objects += New-Object -Type PSObject -Prop #{'InnerText'= $R.InnerText;'href'=$R.href;'Title'=$R.href.split('/')[4]}
}
#output to file
$objects | ConvertTo-HTML -As Table -Fragment | Out-String >> $list_F
You cannot meaningfully use an array as the RHS of an -eq operation (the array will be implicitly stringified, which won't work as intended).
PowerShell has operators -contains and -in to test membership of a value in an array (using -eq on a per-element basis - see this answer for background); therefore:
if ($R.innerText -in $TakeOut) {Continue}
Generally, your code can be streamlined (PSv3+ syntax):
$TakeOut =
"Help",
"Home",
"News",
"Sports",
"Terms of use",
"Travel",
"Video",
"Weather"
#Retrieve website information
$Results = (Invoke-WebRequest -Uri "https://www.msn.com/en-ca/").Links
#Filter and format to new table of values
$objects = foreach($R in $Results) {
if ($R.innerText -in $TakeOut) {Continue}
[pscustomobject #{
InnerText = $R.InnerText
href = $R.href
Title = $R.href.split('/')[4]
}
}
#output to file
$objects | ConvertTo-HTML -As Table -Fragment >> $list_F
Note the absence of #(...), which is never needed for array literals.
Building an array in a loop with += is slow (and verbose); simply use the foreach statement as an expression, which returns the loop body's outputs as an array.
[pscustomobject] #{ ... } is PSv3+ syntactic sugar for constructing custom objects; in addition to being faster than a New-Object call, it has the added advantage of preserving property order.
You could write the whole thing as a single pipeline:
#Retrieve website information
(Invoke-WebRequest -Uri "https://www.msn.com/en-ca/").Links | ForEach-Object {
#Filter and format to new table of values
if ($_.innerText -in $TakeOut) {return}
[pscustomobject #{
InnerText = $_.InnerText
href = $_.href
Title = $_.href.split('/')[4]
}
} | ConvertTo-HTML -As Table -Fragment >> $list_F
Note the need to use return instead of continue to move on to the next input.

PowerShell TFS REST-API object loop advise

I have a piece of code that i managed to get working, but i feel that it can be written a lot easier. Im new with PowerShell and am trying to understand it better. I have a double foreach below to get the key and value out of the PSCustomObject that comes out of the TFS REST-API call.
For some reason im doing 2 loops, but i dont understand why this is required.
A sample of the contents of $nameCap.userCapabilities is
Name1 Name2
----- -----
Value1 Value2
So basically i want to loop over the "name/value pairs" and get their values.
What can i do better ?
$uri = "$tfsUri/_apis/distributedtask/pools/$global:agentPoolId/agents?api-version=3.0-preview&includeCapabilities=true"
$result = (Invoke-RestMethod -Uri $uri -Method Get -ContentType "application/json" -UseDefaultCredentials).value | select name, userCapabilities, systemCapabilities
#Loop over all agents and their capablities
foreach ($nameCap in $result)
{
$capabilityNamesList = New-Object System.Collections.ArrayList
#Loop over all userCapabilities and store their names
#($nameCap.userCapabilities) | %{
$current_Cap = $_
$req_cap_exists = $false
Get-Member -MemberType Properties -InputObject $current_Cap | %{
$temp_NAME = $_.Name
$temp_Value = Select-Object -InputObject $current_Cap -ExpandProperty $_.Name
[void]$capabilityNamesList.Add($temp_NAME)
}
}
}
I mean if you just need the Name and value, like userCapabilities, then just select for it.
so:
$result | select Name,userCapabilites
And if it doesn't give you a table automatically, then | ft -force

PowerShell export multiple objects as csv

I have two columns of data the first is a string array the second is actually an object. I am looking for a simple way of exporting this as a csv. I have a version with a foreach loop that builds each string up but it seams like over kill. I have been trying to use select and select object to get it out somehow. Note I am just a beginner at powershell so I may be missing something.
My first attempt:
$data | Select-Object -ExpandProperty reports | Select -ExpandProperty data | Select -ExpandProperty rows | Format-Table $_.dimensions
Results in:
dimensions metrics
---------- -------
{New Visitor, "Mozilla} {#{values=System.Object[]}}
My second one went as far as looping
foreach ($report in $data.reports) {
"Rows:" + $report.data.rows.Count
foreach ($row in $report.data.rows) {
$output = ""
foreach($dim in $row.dimensions) {
$output += $dim + $seporator
}
foreach($met in $row.metrics) {
foreach($v in $met.values) {
$output += $v + $seporator
}
}
#| Out-File -Append C:\Users\linda_l\Documents\debug.txt
$output
}
}
There is a potential for a lot of data here so I would really like to avoid the string building solution.
Note: Data comes from the Google Analytics reporting api
$data = Invoke-RestMethod -ContentType 'application/json' -Uri "https://analyticsreporting.googleapis.com/v4/reports:batchGet?access_token=$($token.access_token)" -Method POST -Body $analyticsRequest
reports : {#{columnHeader=; data=}}
From comment:
$data | Export-Csv -Path "C:\Users\linda_l\Documents\debug2.csv"
System.Management.Automation.PSCustomObject reports System.Object[]
Optimal output csv
New Visitor,S40 Ovi Browser,1,2
New Visitor,Safari,3,4
Note its up on Github steps for getting a refreshtoken
Data is coming from the Google analytics reporting API