I have a basic bat script that uses [curl]( https://curl.haxx.se/). The script takes values from a txt document named location_ids.txt (this file is found in the same folder as the script). I have set it up to check the location_id with 3 different urls. It works well. However, it is very slow! since batch files aren't threaded, and each command will block until it completes. I am aware this can be done with more ease and speed using a powershell script (windows environment) using Invoke-RestMethod. How can I replicate the below in ps? I would like to make the curl calls simultaneous.
#echo off
setlocal enabledelayedexpansion
set OUTPUT_FILE=%DATE:/=-%#%TIME::=-%
set OUTPUT_FILE=file_%OUTPUT_FILE: =%.html
for /f %%i in (location_ids.txt) do (
set LOCATION_ID=%%i
echo ^<h2 style='color:green;'^>!LOCATION_ID::=!^</h2^> >>%OUTPUT_FILE%
curl -k -H "application/x-www-form-urlencoded" -X GET -d "id=!LOCATION_ID::=!" http://localhost:5000/location_site1 >>%OUTPUT_FILE%
curl -k -H "application/x-www-form-urlencoded" -X GET -d "id=!LOCATION_ID::=!" http://localhost:5000/location_site2 >>%OUTPUT_FILE%
curl -k -H "application/x-www-form-urlencoded" -X GET -d "id=!LOCATION_ID::=!" http://localhost:5000/location_site3 >>%OUTPUT_FILE%
echo ^<br^>^<br^> >>%OUTPUT_FILE%
)
EDIT:
My attempt to run multiple web server calls to http://localhost:5000/location_site1 that run simultaneously using scriptblock. The below is not outputting any results.
$runRoutePost =
{ param([string]$id, [string]$fileLocation)
Write-Host "Accessing site for $id";
$ResponseData = New-Object PSObject;
$webclient = new-object System.Net.WebClient;
$apiParams = "id=$_";
$ResponseData = $webclient.UploadString("http://localhost:5000/location_site1",$apiParams) |Add-Content $fileLocation;
}
Get-Content location_ids.txt | ForEach-Object {
Start-Job -ScriptBlock $runRoutePost -ArgumentList $_, $LOG_FILE
}
To convert your example, you really just need to make a request to the url and specify the location id as a query string parameter. The example below uses string interpolation to set the value of the id parameter. The $_ variable is the current item that is being enumerated within the ForEach-Object script block.
$outputFile = # not sure of your date time format
Get-Content location_ids.txt | ForEach-Object {
Add-Content $outputFile "<h2 style=`"color:green`">$_</h2>"
Invoke-RestMethod -Uri "http://localhost:5000/location_site1?id=$_" | Add-Content $outputFile
Invoke-RestMethod -Uri "http://localhost:5000/location_site2?id=$_" | Add-Content $outputFile
Invoke-RestMethod -Uri "http://localhost:5000/location_site3?id=$_" | Add-Content $outputFile
Add-Content $outputFile "<br><br>"
}
For a GET request you do not need to specify the content-type or method. However, if you need to for other scripts you can use the -ContentType and/or -Method parameters.
Invoke-RestMethod -Method GET -ContentType application/x-www-form-urlencoded -Uri "http://localhost:5000/location_site3?id=$_"
More documentation can be found by running this:
get-help Invoke-RestMethod -full
Since you have a restriction of using PowerShell v2, you can use the .NET WebClient type.
$web = new-object System.Net.WebClient
Get-Content location_ids.txt | ForEach-Object {
Add-Content $outputFile "<h2 style=`"color:green`">$_</h2>"
$web.DownloadString("http://localhost:5000/location_site1?id=$_") | Add-Content $outputFile
$web.DownloadString("http://localhost:5000/location_site2?id=$_") | Add-Content $outputFile
$web.DownloadString("http://localhost:5000/location_site3?id=$_") | Add-Content $outputFile
Add-Content $outputFile "<br><br>"
}
If instead you want to send a POST request using WebClient, the UploadString method can be used. However, in this case I'm not sure of how to set the Content-Type header.
$web.UploadString("http://localhost:5000/location_site1","id=$_") | Add-Content $outputFile
Update in response to edit
To run these jobs in parallel and collect the results, you need to wait for all the jobs to complete using Wait-Job and then extract the results using Receive-Job.
$runRoutePost = {
param([string]$id)
Write-Host "Accessing site for $id"
$webclient = new-object System.Net.WebClient
$webclient.UploadString("http://localhost:5000/location_site1","id=$id")
}
$Jobs = Get-Content location_ids.txt | ForEach-Object {
Start-Job -ScriptBlock $runRoutePost -ArgumentList $_
}
Wait-Job -Job $Jobs
Receive-Job -Job $Jobs | Add-Content $LOG_FILE
Related
I am attempting to pull the URL once an InvokeWebRequest has been called in my script. I wish for it do this when the HTTP Status Code is 200. So far, I have got it to tell the Status Code, and give the re-directed URL is 301/302 is the Status Code. Also an Error Message when the Status Code is 4xx//5xx.
I deal with a lot of URLs on clients' ecommerce sites, so need know what the final URL is after a end-user clicks on one of the URLs - essentially something like the query strings being dropped or something like that, on Status Code 200. As mentioned before, I already have 3xx/4xx/5xx reporting.
Attempted -
Reading the page and pulling the URL from the Header
Searching for the 'href' and pulling it
I wish for the CSV out to have the following output -
Expected CSV Output
Actual CSV Output (Row 3)
The "Landing Page" field is what I am trying to fill - the rest are working.
Script below -
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$urlInput = "C:\Users\User\Desktop\testurls.txt"
$csvOutput = "C:\Users\User\Desktop\testurl_result.csv"
#$totalUrls = Get-Content $urlInput | Measure-Object –Line
#Write-Host "Total URLs = $totalUrls"
$results = Test-Path "C:\Users\User\Desktop\testurl_result.csv"
If ($results -eq $false) {
Remove-Item "testurl_result.csv" -Force
}
Get-Content $urlInput | ForEach-Object { $uri = $_;
try
{
Start-Sleep 5
Invoke-WebRequest -Uri $uri -Method HEAD -MaximumRedirection 0 -ErrorAction SilentlyContinue
$link = Invoke-WebRequest -Uri $uri -Method HEAD -MaximumRedirection 0 -ErrorAction SilentlyContinue
}
catch
{
New-Object -TypeName psobject -Property #{Error = $_}
}
} | Select-Object #{Name='Click URL'; Expression={$uri}}, StatusCode, #{Name='Path'; Expression={([uri]$uri).AbsolutePath}}, #{Name='UTMs'; Expression={([uri]$uri).Query}}, #{Name='RedirectTo';Expression={$_.Headers["Location"]}}, #{Name='Landing Page';Expression={$link.Links | select -ExpandProperty href}}, Error | Export-Csv $csvOutput -NoTypeInformation
Thanks.
Can any one help on how to get start time, end time , duration and execute an API using powershell script
example
Need to execute below sample API
Get current UTC time in this format "2021-05-22T12:15:27.000027" and store in this variable
StartTimeInUTC = 2021-05-22T12:15:27.000027
invoke below API
https:test.com/${StartTimeInUTC}
Below code shows how you can call API in powershell.
$StartTimeInUTC = Get-Date -Format o | ForEach-Object { $_ -replace ":", "." }
Write-Host $StartTimeInUTC
$url = "https://test.com/${StartTimeInUTC}"
Write-Host $url
$webrequest = Invoke-WebRequest -Uri $url -SessionVariable websession -UseBasicParsing
$EndTimeInUTC = Get-Date -Format o | ForEach-Object { $_ -replace ":", "." }
Write-Host $EndTimeInUTC
I have a URL health-checking PowerShell script which correctly gets an HTTP 200 status code on most of my intranet sites, but a '0' status code is returned on a small minority of them. The '0' code is an API return rather than from the web site itself, according to my research of questions from others who have written similar URL-checking PowerShell scripts. Thinking this must be a timeout issue, where API returns '0' before the slowly-responding web site returns its 200, I've researched yet more questions about this subject area on SO and implemented a suggestion from someone to insert a timeout in the script. The timeout setting though, no matter how high I set the timeout value, doesn't help. I still get the same '0' "response" code from the same web sites even though those web sites are up and running as checked from any regular web browser. Any thoughts on how I could tweak the timeout setting in the script below in order to get the correct 200 response code?
The Script:
$URLListFile = "C:\Users\Admin1\Documents\Scripts\URL Check\URL_Check.txt"
$URLList = Get-Content $URLListFile -ErrorAction SilentlyContinue
#if((test-path $reportpath) -like $false)
#{
#new-item $reportpath -type file
#}
#For every URL in the list
$result = foreach($Uri in $URLList) {
try{
#For proxy systems
[System.Net.WebRequest]::DefaultWebProxy = [System.Net.WebRequest]::GetSystemWebProxy()
[System.Net.WebRequest]::DefaultWebProxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
#Web request
$req = [system.Net.WebRequest]::Create($uri)
$req.Timeout=5000
$res = $req.GetResponse()
}
catch {
#Err handling
$res = $_.Exception.Response
}
$req = $null
#Getting HTTP status code
$int = [int]$res.StatusCode
# output a formatted string to capture in variable $result
"$int - $uri"
#Disposing response if available
if($res){
$res.Dispose()
}
}
# output on screen
$result
#output to log file
$result | Set-Content -Path "C:\Users\Admin1\Documents\Scripts\z_Logs\URL_Check\URL_Check_log.txt" -Force
Current output:
200 - http://192.168.1.1/
200 - http://192.168.1.2/
200 - http://192.168.1.250/config/authentication_page.htm
0 - https://192.168.1.50/
200 - http://app1-vip-http.dev.local/
0 - https://CA/certsrv/Default.asp
Perhaps using PowerShell cmdlet Invoke-WebRequest works better for you. It has many more parameters and switches to play around with like ProxyUseDefaultCredentials and DisableKeepAlive
$pathIn = "C:\Users\Admin1\Documents\Scripts\URL Check\URL_Check.txt"
$pathOut = "C:\Users\Admin1\Documents\Scripts\z_Logs\URL_Check\URL_Check_log.txt"
$URLList = Get-Content -Path $pathIn
$result = foreach ($uri in $URLList) {
try{
$res = Invoke-WebRequest -Uri $uri -UseDefaultCredentials -UseBasicParsing -Method Head -TimeoutSec 5 -ErrorAction Stop
$status = [int]$res.StatusCode
}
catch {
$status = [int]$_.Exception.Response.StatusCode.value__
}
# output a formatted string to capture in variable $result
"$status - $uri"
}
# output on screen
$result
#output to log file
$result | Set-Content -Path $pathOut -Force
I am trying to download the Total counts by date for all King County excel file using a script that will be run later on using task manager. I am stuck due to the link not being static and its naming convention will most likely change in the next few months.
Here's the code that I've written so far:
#echo off
::Script to download COVID-19 Data
echo "Downloading Total counts by date for all King County"
powershell -Command "Invoke-WebRequest https://www.kingcounty.gov/depts/health/covid-19/data/~/media/depts/health/communicable-diseases/documents/C19/data/covid-data-daily-counts-sept-14.ashx -Outfile CovidData.xlsx
echo "Download has been successful!"
cls
pause
I was wondering if there's a way to add a wild card like "*" in the invoke-webrequest to ignore the "sept-14" part of the link.
Link: https://www.kingcounty.gov/depts/health/covid-19/data/daily-summary.aspx
Link that needs a script to auto download with task manager (Total counts by date for all King County): https://www.kingcounty.gov/depts/health/covid-19/data/~/media/depts/health/communicable-diseases/documents/C19/data/covid-data-daily-counts-sept-14.ashx
I created and tested Powershell Script on my side with Windows Powershell ISE and it works 5/5, hope that will work too for you !
$start_time = Get-Date
$url = "https://www.kingcounty.gov/depts/health/covid-19/data/daily-summary.aspx"
$xlFile = "E:\Test\CovidData.xlsx"
$http_request = New-Object -ComObject Microsoft.XMLHTTP
$http_request.open('GET', $url, $false)
#Sending the request
$http_request.send()
$Contents = $http_request.ResponseText
$pattern = "([\w\-\.,#?^=%&/~\+#]*[\w\-\#?^=%&/~\+#])(\.ashx)"
$Links = $Contents | Select-String $pattern -AllMatches | ForEach-Object {$_.Matches.Value} | sort -unique
$Filter = $Links -match "data-daily"
$MyUrlFile = "https://www.kingcounty.gov/depts/health/covid-19/data/" + $Filter
$MyUrlFile
Invoke-WebRequest "$MyUrlFile" -Outfile $xlFile
if (Test-Path $xlFile) { Start $xlFile }
Write-Output "Running Script Time taken is : $((Get-Date).Subtract($start_time).Seconds) second(s)"
Hi I have a script to automate some tasks, running in powershell core v.7.+.
In one of these scripts when I run the command inside the ps1 file and the special characters returned is encoded and I can't decode to right format, below the command used to do this and the return:
// the variable $script is my command its a ask-cli command to work in alexa
$model = pwsh -Command $script
/*
* when I running the script from here the special characters returned is these:
* "nächste",
* "nächstgelegene"
*/
But when I run the same command directly in the terminal the strings returned are:
/*
* "nächste",
* "nächstgelegene"
*/
I would like to know how can I run the command inside the file without encode the return.
I already tried some things as:
$encoding = [System.Text.Encoding]::Unicode
$model = pwsh -Command $script
Write-Output $model
$model = $encoding.GetBytes($model)
$model = $encoding.GetString($model)
But don't work as expected, I don't know more how I can to this, if someone could help me with this I appreciate too much.
Try returning the string as bytes and then decode it from the place you are calling the function pwsh. This would preserve it from any changes. What you're doing is converting it into bytes after receiving it then returning it to string.
Below my script:
(Get-Content "$currentPath\skill-package\skill.json" -Raw | ConvertFrom-Json).manifest.publishingInformation.locales.PSObject.Properties | ForEach-Object {
$lang = $_.Name
Write-Output "Profile: $profile skill-id: $skillId language: $lang"
$script = "ask smapi get-interaction-model -l $lang -s $skillId -p $profile -g $env"
Write-Output 'Running script'
Write-Warning $script
# $encoding = [System.Text.Encoding]::ASCII
$model = pwsh -Command $script
Write-Output $model
$model = $model
| ConvertFrom-Json -Depth 100
| Select-Object * -ExcludeProperty version
| ConvertTo-Json -Depth 100
# out-file "$file$lang.json" -InputObject $model -Encoding ascii
Write-Output "New model saved locally $file$lang.json"
}
Write-Warning 'Finished!!!'
(Get-Content "$currentPath\skill-package\skill.json" -Raw | ConvertFrom-Json).manifest.publishingInformation.locales.PSObject.Properties | ForEach-Object {
$lang = $_.Name
Write-Output "Profile: $profile skill-id: $skillId language: $lang"
$script = "`$command = ask smapi get-interaction-model -l $lang -s $skillId -p $profile -g $env;`$command = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes(Invoke-Expression `$command));`$command"
Write-Output 'Running script'
Write-Warning $script
# $encoding = [System.Text.Encoding]::ASCII
$model = pwsh -Command $script
$model = Text.Encoding::Unicode.GetString([Convert]::FromBase64String($model))
Write-Output $model
$model = $model
| ConvertFrom-Json -Depth 100
| Select-Object * -ExcludeProperty version
| ConvertTo-Json -Depth 100
# out-file "$file$lang.json" -InputObject $model -Encoding ascii
Write-Output "New model saved locally $file$lang.json"
}
Write-Warning 'Finished!!!'
After many researches, I could find something more easiest to solve my problem.
Powershell has by default a different output encoder used in these cases, and the only thing I need to do it's change it.
I used the command:
$OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = New-Object System.Text.UTF8Encoding
I find this question explaining how this work and this help a lot, for more question please check this answer.