Adding Powershell jobs to retrieve in parallel from paginated rest API - rest

Question

I've spent perhaps over 30 hours trying to add various methods to implement parallel processing to my API call. The most obvious would be to place the code into a job, but I'm having no luck. Any ideas?
Start-Transcript -Path "$PSScriptRoot\Errorlog.txt"
$WondeObjectsArray = Import-CSV $PSScriptRoot\WondeID.csv
$EndpointObjectArray = Import-CSV $PSScriptRoot\Endpoints.csv
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("REDACTED", "REDACTED")
# School loop
Foreach ($object in $WondeObjectsArray) {
$SchoolName = $object.School
$TrustName = $object.Trust
$WondeID = $object."Wonde ID"
# Create data structure
If(!(test-path "$PSScriptRoot\$TrustName"))
{New-Item -Path "$PSScriptRoot\$TrustName" -ItemType Directory}
If(!(test-path "$PSScriptRoot\$TrustName\$SchoolName"))
{New-Item -Path "$PSScriptRoot\$TrustName\$SchoolName" -ItemType Directory}
# Endpoint request loop
foreach($Eobject in $EndpointObjectArray){
$JsonName = $Eobject.JsonName
$Table = $Eobject.Table
$Method = $Eobject.Url_Method
# First response
$response = Invoke-RestMethod "https://api.wonde.com/v1.0/schools/$WondeID/$Table$Method&$Page" -Method 'GET' -Headers $headers -Body $body
$concat = $response.data
#Pagination loop
While ($response.meta.pagination.next){
$response = Invoke-RestMethod $response.meta.Pagination.next -Method 'GET' -Headers $headers -Body $Body
$concat = $concat + $response.data
} #pagination loop end
# Concatenate completed request
$concat | ConvertTo-Json | Out-File "$PSScriptRoot\$TrustName\$SchoolName\$JsonName.json"
} # Endpoint request loop end
} # School loop end
Stop-Transcript

Answer 1

Never mind, I figured it out. I turned the code that I wanted to run in parallel into a scriptblock, Passed it the parameters by using param () And in the start job line I included my variables in -ArgumentList.
My issue was actually that when you start a new job it changes the script file path to C:/, when I was passing $PSScriptRoot rather than the default file location.
Start-Transcript -Path "$PSScriptRoot\Errorlog.txt"
$PathRoot = $PSScriptRoot
$WondeObjectsArray = Import-CSV $PSScriptRoot\WondeID.csv
$EndpointObjectArray = Import-CSV $PSScriptRoot\Endpoints.csv
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("REDACTED", "REDACTED")
# School loop
Foreach ($object in $WondeObjectsArray) {
$SchoolName = $object.School
$TrustName = $object.Trust
$WondeID = $object."Wonde ID"
# Create data structure
If(!(test-path "$PSScriptRoot\$TrustName"))
{New-Item -Path "$PSScriptRoot\$TrustName" -ItemType Directory}
If(!(test-path "$PSScriptRoot\$TrustName\$SchoolName"))
{New-Item -Path "$PSScriptRoot\$TrustName\$SchoolName" -ItemType Directory}
# Endpoint request loop
foreach ($Eobject in $EndpointObjectArray){
$JsonName = $Eobject.JsonName
$Table = $Eobject.Table
$Method = $Eobject.Url_Method
$scriptblock = {
param ($WondeID, $Table, $Method, $headers, $trustName, $SchoolName, $JsonName, $PathRoot)
write-host "$TrustName : $SchoolName : $JsonName is complete!"
# First response
$response = Invoke-RestMethod "https://api.wonde.com/v1.0/schools/$WondeID/$Table$Method&$Page" -Method 'GET' -Headers $headers -Body $body
$concat = $response.data
#Pagination loop
While ($response.meta.pagination.next){
$response = Invoke-RestMethod $response.meta.Pagination.next -Method 'GET' -Headers $headers -Body $Body
$concat = $concat + $response.data
} #pagination loop end
# Concatenate completed request
$concat | ConvertTo-Json | Out-File "$PathRoot\$TrustName\$SchoolName\$JsonName.json"
}
Start-Job $scriptblock -Name "$SchoolName&$JsonName" -ArgumentList $WondeID, $Table, $Method, $headers, $trustName, $SchoolName, $JsonName, $PathRoot
} # Endpoint request loop end
Get-Job | Wait-Job | Receive-Job
Get-Job | Remove-Job
} # School loop end
Stop-Transcript