I have a piece of code that i managed to get working, but i feel that it can be written a lot easier. Im new with PowerShell and am trying to understand it better. I have a double foreach below to get the key and value out of the PSCustomObject that comes out of the TFS REST-API call.
For some reason im doing 2 loops, but i dont understand why this is required.
A sample of the contents of $nameCap.userCapabilities is
Name1 Name2
----- -----
Value1 Value2
So basically i want to loop over the "name/value pairs" and get their values.
What can i do better ?
$uri = "$tfsUri/_apis/distributedtask/pools/$global:agentPoolId/agents?api-version=3.0-preview&includeCapabilities=true"
$result = (Invoke-RestMethod -Uri $uri -Method Get -ContentType "application/json" -UseDefaultCredentials).value | select name, userCapabilities, systemCapabilities
#Loop over all agents and their capablities
foreach ($nameCap in $result)
{
$capabilityNamesList = New-Object System.Collections.ArrayList
#Loop over all userCapabilities and store their names
#($nameCap.userCapabilities) | %{
$current_Cap = $_
$req_cap_exists = $false
Get-Member -MemberType Properties -InputObject $current_Cap | %{
$temp_NAME = $_.Name
$temp_Value = Select-Object -InputObject $current_Cap -ExpandProperty $_.Name
[void]$capabilityNamesList.Add($temp_NAME)
}
}
}
I mean if you just need the Name and value, like userCapabilities, then just select for it.
so:
$result | select Name,userCapabilites
And if it doesn't give you a table automatically, then | ft -force
Related
I want to get download URL of the last version of GIMP from it's site ,I wrote a script but it returns the link name I do not know how to get the value
$web = Invoke-WebRequest -Uri "https://download.gimp.org/pub/gimp/v2.10/windows/"
$web.Links | Where-Object href -like '*exe' | select -Last 1 | select -expand href
the above code returne link name (gimp-2.10.32-setup.exe)
but I need the value ("https://download.gimp.org/pub/gimp/v2.10/windows/gimp-2.10.32-setup.exe")
can someone guide me how to do it
You know that the url presented is relative.
Just append the root part of the URL yourself.
$Uri = 'https://download.gimp.org/pub/gimp/v2.10/windows/'
$web = Invoke-WebRequest -Uri $uri
$ExeRelLink = $web.Links | Where-Object href -like '*exe' | select -Last 1 -expand href
# Here is your download link.
$DownloadLink = $Uri + $ExeRelLink
Additional Note
You can combine the -Last and -Expand from your 2 select statements into 1.
There are several downloads sites with exactly the same or very similar layout to this GIMP page, including many Apache projects like Tomcat and ActiveMQ. I had written a little function to parse these and other pages in the past, and interestingly it also worked for this GIMP page. I thought it was worth sharing as such.
Function Extract-FilenameFromWebsite {
[cmdletbinding()]
Param(
[parameter(Position=0,ValueFromPipeline)]
$Url
)
begin{
$pattern = '<a href.+">(?<FileName>.+?\..+?)</a>\s+(?<Date>\d+-.+?)\s{2,}(?<Size>\d+\w)?'
}
process{
$website = Invoke-WebRequest $Url -UseBasicParsing
switch -Regex ($website.Content -split '\r?\n'){
$pattern {
[PSCustomObject]#{
FileName = $matches.FileName
URL = '{0}{1}' -f $Url,$matches.FileName
LastModified = [datetime]$matches.Date
Size = $matches.Size
}
}
}
}
}
It's assumed the site passed in has a trailing slash. If you want to account for either, you can add this simple line to the process block.
if($Url -notmatch '/$'){$Url = "$Url/"}
To get the latest version, call the function like this
$url = 'https://download.gimp.org/pub/gimp/v2.10/windows/'
$latest = Extract-FilenameFromWebsite -Url $Url | Where-Object filename -like '*exe' |
Sort-Object LastModified | Select-Object -Last 1
$latest.url
Or you could expand the property while retrieving
$url = 'https://download.gimp.org/pub/gimp/v2.10/windows/'
$latesturl = Extract-FilenameFromWebsite -Url $Url | Where-Object filename -like '*exe' |
Sort-Object LastModified | Select-Object -Last 1 -ExpandProperty URL
$latesturl
I have some working code that basically queries 2 different Graph API endpoints, then searches for a match in the User Principal Name column, and inserts the extension_335d4df9847945fbaa472c8b8fbb5d75_employeeNumber column and values to the exported csv (Thanks to the user #PMental for this solution) This column derives from attribute that was recently extended from our on premises AD.
This code works perfectly fine, however if I try to parallelize it, I get no results in the extension_335d4df9847945fbaa472c8b8fbb5d75_employeeNumber column.
Is this because once it is being parallelized, I'm not able to share variables between the parallel processes? If so, how on earth do I accomplish this?
Code below - if you remove the -Parallel, it works fine:
$graphApiUri = "https://graph.microsoft.com/v1.0/reports/getOffice365ActiveUserDetail(period='D90')"
$Uri = "https://graph.microsoft.com/v1.0/users?`$select=userPrincipalName,extension_335d4df9847945fbaa472c8b8fbb5d75_employeeNumber"
$O365Report = Invoke-RestMethod -Method Get -Uri $graphApiUri -Headers $headerParams | ConvertFrom-Csv
# If the result is more than 999, we need to read the #odata.nextLink to show more than one side of users
$UserDetails = while (-not [string]::IsNullOrEmpty($uri)) {
# API Call
$apiCall = try {
Invoke-RestMethod -Headers $headerParams -Uri $uri -Method Get
}
catch {
$errorMessage = $_.ErrorDetails.Message | ConvertFrom-Json
}
$uri = $null
if ($apiCall) {
# Check if any data is left
$uri = $apiCall.'#odata.nextLink'
$apiCall
}
}
Write-Output "Matching UPN to employeeNumber..."
$O365Report | ForEach-Object -Parallel {
$CurrentEmpNumber = $UserDetails.value |
Where-Object userPrincipalName -eq $_.'User Principal Name' |
Select-Object -ExpandProperty extension_335d4df9847945fbaa472c8b8fbb5d75_employeeNumber -ErrorAction SilentlyContinue
$_ | Add-Member -MemberType NoteProperty -Name extension_335d4df9847945fbaa472c8b8fbb5d75_employeeNumber -Value $CurrentEmpNumber
}
$O365Report | Export-Csv $ReportCSV -NoTypeInformation
Write-Output "Report saved to $ReportCSV."
When inside of a ForEach-Object -Parallel script block, and you are trying to reference variables which were created outside of it, you need to preface the variable name with using: so it would be $using:UserDetails
Examples:
Returns nothing because $test isn't accessible within the scope of the parallel script block:
$test = 1;
0..5 | % -Parallel { $test; };
Returns the value of $test five times because by using $using:test you are now able to see its value:
$test = 1;
0..5 | % -Parallel { $using:test; };
From documenation:
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object?view=powershell-7.1
The ForEach-Object -Parallel parameter set runs script blocks in parallel on separate process threads. The $using: keyword allows passing variable references from the cmdlet invocation thread to each running script block thread. Since the script blocks run in different threads, the object variables passed by reference must be used safely. Generally it is safe to read from referenced objects that don't change. But if the object state is being modified then you must used thread safe objects, such as .Net System.Collection.Concurrent types (See Example 11).
Personal note:
I would also recommend using -ThrottleLimit to limit its max degrees of paralellism. The default is 5, but you may want more or less than that depending on testing.
I'm trying to add a Totals line to the array $response_table.
The line should be Totals = 56 (number), for example.
I've tried $response_table += #{Name="Total"; Count=55};, it adds a line with rubbish data.
Can you help with how to add this?
Code is below
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12
$current_month = (Get-Date).month;
$current_year = (Get-Date).year;
$response_table=#();
$what_year=2019
$month_count=12
$current_month, $total_year, $current_date_interval;
$total_monthly=#(); # empty array, will be populated with values from the API
#$numbers += 5, 2, 3; # to add values to array
if ($what_year -eq $current_year)
{
$month_count = $current_month-1;
}
for ($current_month=1; $current_month -le $month_count; $current_month++)
{
$current_date_interval = -join($what_year, "-", $current_month);
$uri="https://data.police.uk/api/crimes-street/bicycle-theft?poly=53.950624,-1.059234:53.951301,-1.049181:53.947361,-1.043420:53.950333,-1.030671:53.952997,-1.016427:53.950189,-1.013653:53.929487,-1.042286:53.942512,-1.054948:53.941936,-1.057172:53.944481,-1.060155s8&date="+$current_date_interval;
$response = Invoke-RestMethod -Uri $uri -Method Get -Headers $headers
$response_table += $response | Group month |Select -Property name,count
Write-Host $response_table;
$response | Group month |Select -Property name,count
$total_year += $response.count;
}
Write-Host ($response_table|Measure-object -Property count -sum)
$response_table += #{Name="Total"; Count=55};
# does not work
$response_table | export-csv "list.csv" -notypeinformation
#add-content "list.csv" "Total,$total_year"
Write-Host "Yearly total"$total_year;
As AdminOfThings indicates, you're mistakenly adding a hashtable to the existing array of custom objects (that Select-Object outputs).
To construct a custom object in PSv3+, simply place [pscustomobject] before a hashtable literal, which in your case means:
$response_table += [pscustomobject] #{Name="Total"; Count=55}
Now your Export-Csv command should work as intended.
As an aside: Up to PowerShell 6.x, Export-Csv only (meaningfully) supports custom objects as input, not also hashtables. v7 adds support for hashtables.
Generally avoid using += for appending to arrays:
Arrays are fixed-size data structures, so what PowerShell must do when you "append to" an array with += is to create a new array behind the scenes every time, which is quite inefficient in a loop.
While using an efficiently in-place extensible data structure such as [System.Collections.ArrayList] or [System.Collections.Generic.List[object]] is the next best thing, the simplest and fastest approach is to simply let PowerShell collect multiple output objects in an array ([object[]]) for you, by assigning the entire loop as an expression to a variable:
[array] $response_table = for ($current_month=1; $current_month -le $month_count; $current_month++)
{
$current_date_interval = -join($what_year, "-", $current_month);
$uri="https://data.police.uk/api/crimes-street/bicycle-theft?poly=53.950624,-1.059234:53.951301,-1.049181:53.947361,-1.043420:53.950333,-1.030671:53.952997,-1.016427:53.950189,-1.013653:53.929487,-1.042286:53.942512,-1.054948:53.941936,-1.057172:53.944481,-1.060155s8&date="+$current_date_interval;
$response = Invoke-RestMethod -Uri $uri -Method Get -Headers $headers
# *Output* the result of this pipeline, and PowerShell will
# collect all outputs for you
$response | Group month | Select -Property name,count
$total_year += $response.count;
}
# It's fine to use += to add the total, because you're only using it *once*.
$response_table += [pscustomobject] #{Name="Total"; Count=55}
Note: The [array] type constraint ensures that $response_table becomes an array even if the loop happens to have just 1 iteration.
I have two columns of data the first is a string array the second is actually an object. I am looking for a simple way of exporting this as a csv. I have a version with a foreach loop that builds each string up but it seams like over kill. I have been trying to use select and select object to get it out somehow. Note I am just a beginner at powershell so I may be missing something.
My first attempt:
$data | Select-Object -ExpandProperty reports | Select -ExpandProperty data | Select -ExpandProperty rows | Format-Table $_.dimensions
Results in:
dimensions metrics
---------- -------
{New Visitor, "Mozilla} {#{values=System.Object[]}}
My second one went as far as looping
foreach ($report in $data.reports) {
"Rows:" + $report.data.rows.Count
foreach ($row in $report.data.rows) {
$output = ""
foreach($dim in $row.dimensions) {
$output += $dim + $seporator
}
foreach($met in $row.metrics) {
foreach($v in $met.values) {
$output += $v + $seporator
}
}
#| Out-File -Append C:\Users\linda_l\Documents\debug.txt
$output
}
}
There is a potential for a lot of data here so I would really like to avoid the string building solution.
Note: Data comes from the Google Analytics reporting api
$data = Invoke-RestMethod -ContentType 'application/json' -Uri "https://analyticsreporting.googleapis.com/v4/reports:batchGet?access_token=$($token.access_token)" -Method POST -Body $analyticsRequest
reports : {#{columnHeader=; data=}}
From comment:
$data | Export-Csv -Path "C:\Users\linda_l\Documents\debug2.csv"
System.Management.Automation.PSCustomObject reports System.Object[]
Optimal output csv
New Visitor,S40 Ovi Browser,1,2
New Visitor,Safari,3,4
Note its up on Github steps for getting a refreshtoken
Data is coming from the Google analytics reporting API
I have a very large JSON response for employees that I am trying to get into table format, export to CSV and eventually insert into SQL Server. I was able to determine how to get all of my variables from the json file, however now I am getting all of my values inserted on one row for each column instead of a new row for each employee.
Also, when I export to CSV the value turns into System.Object[].
$json1 = Invoke-webRequest -Uri $Workeruri -Certificate $cert -Headers $WorkerHeader | convertfrom-json
$table = [PSCustomObject] #{
associateOID = $json1.workers.associateOID
workerID = $json1.workers.workerID.idValue
GivenName = $json1.workers.person.legalName.givenName
MiddleName = $json1.workers.person.legalName.middleName
FamilyName1 = $json.workers.person.legalName.familyName1
} |format-table -autosize
$table | export-csv $filepath -NoTypeInformation
The columns are a small sample, there are actually probably 100 columns. However, my response returns like this:
associateOID workerID givenName
------------ -------- ---------
{1,2,3,4,5...} {a,b,c,d,e...} {Lebron James, Micheal Jordan, Steph Curry...}
I would like it to return:
associateOID workerID givenName
------------ -------- ---------
1 A Lebron James
2 B Micheal Jordan
3 C Steph Curry
Also, when exporting to CSV the response has the correct columns, but all columns return with: System.Object[].
Also, my fields that have ints and dates are not returning data. How can I fix that as well?
I have tried using sort-object, group-object, for-each loops. Nothing has worked.
you can try like this:
$json1 = Invoke-webRequest -Uri $Workeruri -Certificate $cert -Headers $WorkerHeader | ConvertFrom-Json
$table = $json1 | ForEach-Object {
[PSCustomObject] #{
associateOID = $_.workers.associateOID
workerID = $_.workers.workerID.idValue
GivenName = $_.workers.person.legalName.givenName
MiddleName = $_.workers.person.legalName.middleName
FamilyName1 = $_.workers.person.legalName.familyName1
}
}
$table | Export-Csv $filepath -NoTypeInformation
$table | Format-Table -AutoSize
Your snippet takes all the values for each column and stores them in a single object instead of iterating on the object collection converted from JSON.
Also, once you use Format-Table, data is formatted for display but not usable in the pipeline anymore. That's why I've separated display on screen and CSV export.
#sodawillow almost had it, assuming that json1.workers is the list of objects that contains your workers.
$json1 = Invoke-webRequest -Uri $Workeruri -Certificate $cert -Headers $WorkerHeader | ConvertFrom-Json
$table = $json1.workers | ForEach-Object {
[PSCustomObject] #{
associateOID = $_.associateOID
workerID = $_.workerID.idValue
GivenName = $_.person.legalName.givenName
MiddleName = $_.person.legalName.middleName
FamilyName1 = $_.person.legalName.familyName1
}
}
$table | Export-Csv $filepath -NoTypeInformation
$table | Format-Table -AutoSize