Wonder if anyone can help. I'm trying to use powershell to import a list of domains and generate a HTTP request for each of them. Currently I've got this far:
$csvFilename = "C:\Sites\sites.csv"
$csv = Import-Csv $csvFilename -Header #("url") |
foreach {
$HTTP_Request = [System.Net.WebRequest]::Create('$csv')
$HTTP_Response = $HTTP_Request.GetResponse()
$HTTP_Status = [int]$HTTP_Response.StatusCode
If ($HTTP_Status -eq 200) {
Write-Host "Site is OK!"
}
Else {
Write-Host "The Site may be down, please check!"
}
$HTTP_Response.Close()
}
I'm not sure why but the 'Create()' statement won't pick up the URL. Any ideas?
Many thanks
inside the foreach loop, you can access the current element with the automatic variable $_
so replace this line $HTTP_Request = [System.Net.WebRequest]::Create('$csv')
by
$HTTP_Request = [System.Net.WebRequest]::Create("$_")
but before as create method is waiting for a string,you will have to expand your url property (otherwise you get a pscustomObject not a string) :
Import-Csv c:\temp\urls.csv -Header "url" |select -expand url |%{ [System.Net.Webrequest]::Create("$_")}
Related
I am attempting to pull the URL once an InvokeWebRequest has been called in my script. I wish for it do this when the HTTP Status Code is 200. So far, I have got it to tell the Status Code, and give the re-directed URL is 301/302 is the Status Code. Also an Error Message when the Status Code is 4xx//5xx.
I deal with a lot of URLs on clients' ecommerce sites, so need know what the final URL is after a end-user clicks on one of the URLs - essentially something like the query strings being dropped or something like that, on Status Code 200. As mentioned before, I already have 3xx/4xx/5xx reporting.
Attempted -
Reading the page and pulling the URL from the Header
Searching for the 'href' and pulling it
I wish for the CSV out to have the following output -
Expected CSV Output
Actual CSV Output (Row 3)
The "Landing Page" field is what I am trying to fill - the rest are working.
Script below -
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$urlInput = "C:\Users\User\Desktop\testurls.txt"
$csvOutput = "C:\Users\User\Desktop\testurl_result.csv"
#$totalUrls = Get-Content $urlInput | Measure-Object –Line
#Write-Host "Total URLs = $totalUrls"
$results = Test-Path "C:\Users\User\Desktop\testurl_result.csv"
If ($results -eq $false) {
Remove-Item "testurl_result.csv" -Force
}
Get-Content $urlInput | ForEach-Object { $uri = $_;
try
{
Start-Sleep 5
Invoke-WebRequest -Uri $uri -Method HEAD -MaximumRedirection 0 -ErrorAction SilentlyContinue
$link = Invoke-WebRequest -Uri $uri -Method HEAD -MaximumRedirection 0 -ErrorAction SilentlyContinue
}
catch
{
New-Object -TypeName psobject -Property #{Error = $_}
}
} | Select-Object #{Name='Click URL'; Expression={$uri}}, StatusCode, #{Name='Path'; Expression={([uri]$uri).AbsolutePath}}, #{Name='UTMs'; Expression={([uri]$uri).Query}}, #{Name='RedirectTo';Expression={$_.Headers["Location"]}}, #{Name='Landing Page';Expression={$link.Links | select -ExpandProperty href}}, Error | Export-Csv $csvOutput -NoTypeInformation
Thanks.
As the title suggests, how do I do an if statement for comparing 2 files? This is what I have so far but even though the files are the same it still keeps prompting me for an update
Add-Type -AssemblyName 'PresentationFramework'
$continue = 'Yes'
Invoke-WebRequest -Uri "https://<url>/file" -OutFile "C:\temp\file"
if ( "C:\temp\file" -ne "C:\app\file" ) {
$caption = "Update Available!"
$message = "There is a new version available. Do you want to update to the latest version?"
$continue = [System.Windows.MessageBox]::Show($message, $caption, 'YesNo');
} else {
Write-Output "You have the latest version!"
}
if ($continue -eq 'Yes') {
Write-Output "Downloading application Please wait..."
// Do something //
} else {
Write-Output "Cancelled"
}
Any help would be greatly appreciated! Thanks
This is the logic I would personally follow, you can re-estructure it later for your need but in very basic steps:
Get the MD5 sum of my current file.
Query the new file with Invoke-WebRequest and using a MemoryStream check if the new hash is the same as the previous one.
If it's not, write that stream to a FileStream.
$ErrorActionPreference = 'Stop'
try {
$previousFile = "C:\app\file"
$previousHash = (Get-FileHash $previousFile -Algorithm MD5).Hash
$webReq = Invoke-WebRequest 'https://<url>/file'
$memStream = [IO.MemoryStream] $webReq.Content
$newHash = (Get-FileHash -InputStream $memStream -Algorithm MD5).Hash
if($previousHash -ne $newHash) {
$file = [IO.File]::Create('path\to\mynewfile.ext')
$memStream.WriteTo($file)
}
}
finally {
$file, $memStream | ForEach-Object Dispose
}
The main idea behind it is to check all in memory so as to avoid the need to download a file if not needed.
Essentially what I'm after is the results of rest API Gateways - Get Datasource Users but retaining the ID (in this example $Line.id from my imported CSV file).
The end result should be a CSV with the following fields -
ID, emailAddress, datasourceAccessRight, displayName, identifier, principalType
I'm new to PowerShell and surprised I got this far but can't figure out this final bit.
Cheers
$webclient=New-Object System.Net.WebClient
$webclient.Proxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
$Dir = "C:\pbi_pro_user_logs\"
Login-PowerBI
$GateWayFile = Import-CSV -Path "C:\pbi_pro_user_logs\Gateway_Detail.csv"
$Output = #()
foreach ($Line in $GateWayFile){
$Item = $Line.id
$url = "https://api.powerbi.com/v1.0/myorg/gateways/HIDDEN/datasources/"+$Item+"/users"
$Output += (Invoke-PowerBIRestMethod -Url $url -Method Get | ConvertFrom-Json)
}
$Result = $Output.value
$Result | Export-Csv $Dir"GateWay_users.csv" -NoTypeInformation
Try this, using a calculated property from Select-Object:
$GateWayFile = Import-CSV -Path "C:\pbi_pro_user_logs\Gateway_Detail.csv"
$Output = Foreach ($Line in $GateWayFile){
$url = "https://api.powerbi.com/v1.0/myorg/gateways/HIDDEN/datasources/"+$Line.id+"/users"
$Item = (Invoke-PowerBIRestMethod -Url $url -Method Get | ConvertFrom-Json)
# output all properties of the item, plus the ID:
$ItemWithID = $Item | Select *,#{l='Id';e={$line.id}}
Write-Output $ItemWithID
}
# This depends on how you want your csv structured, but for example:
$Result = $Output | Select Id,Value
Or, if Value is a whole object that ID should be assigned inside of, then change the selection lines:
$ItemWithID = $Item.Value | Select *,#{l='Id';e={$line.id}}
$Result = $Output
I have a URL health-checking PowerShell script which correctly gets an HTTP 200 status code on most of my intranet sites, but a '0' status code is returned on a small minority of them. The '0' code is an API return rather than from the web site itself, according to my research of questions from others who have written similar URL-checking PowerShell scripts. Thinking this must be a timeout issue, where API returns '0' before the slowly-responding web site returns its 200, I've researched yet more questions about this subject area on SO and implemented a suggestion from someone to insert a timeout in the script. The timeout setting though, no matter how high I set the timeout value, doesn't help. I still get the same '0' "response" code from the same web sites even though those web sites are up and running as checked from any regular web browser. Any thoughts on how I could tweak the timeout setting in the script below in order to get the correct 200 response code?
The Script:
$URLListFile = "C:\Users\Admin1\Documents\Scripts\URL Check\URL_Check.txt"
$URLList = Get-Content $URLListFile -ErrorAction SilentlyContinue
#if((test-path $reportpath) -like $false)
#{
#new-item $reportpath -type file
#}
#For every URL in the list
$result = foreach($Uri in $URLList) {
try{
#For proxy systems
[System.Net.WebRequest]::DefaultWebProxy = [System.Net.WebRequest]::GetSystemWebProxy()
[System.Net.WebRequest]::DefaultWebProxy.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
#Web request
$req = [system.Net.WebRequest]::Create($uri)
$req.Timeout=5000
$res = $req.GetResponse()
}
catch {
#Err handling
$res = $_.Exception.Response
}
$req = $null
#Getting HTTP status code
$int = [int]$res.StatusCode
# output a formatted string to capture in variable $result
"$int - $uri"
#Disposing response if available
if($res){
$res.Dispose()
}
}
# output on screen
$result
#output to log file
$result | Set-Content -Path "C:\Users\Admin1\Documents\Scripts\z_Logs\URL_Check\URL_Check_log.txt" -Force
Current output:
200 - http://192.168.1.1/
200 - http://192.168.1.2/
200 - http://192.168.1.250/config/authentication_page.htm
0 - https://192.168.1.50/
200 - http://app1-vip-http.dev.local/
0 - https://CA/certsrv/Default.asp
Perhaps using PowerShell cmdlet Invoke-WebRequest works better for you. It has many more parameters and switches to play around with like ProxyUseDefaultCredentials and DisableKeepAlive
$pathIn = "C:\Users\Admin1\Documents\Scripts\URL Check\URL_Check.txt"
$pathOut = "C:\Users\Admin1\Documents\Scripts\z_Logs\URL_Check\URL_Check_log.txt"
$URLList = Get-Content -Path $pathIn
$result = foreach ($uri in $URLList) {
try{
$res = Invoke-WebRequest -Uri $uri -UseDefaultCredentials -UseBasicParsing -Method Head -TimeoutSec 5 -ErrorAction Stop
$status = [int]$res.StatusCode
}
catch {
$status = [int]$_.Exception.Response.StatusCode.value__
}
# output a formatted string to capture in variable $result
"$status - $uri"
}
# output on screen
$result
#output to log file
$result | Set-Content -Path $pathOut -Force
function Get-Data(){
[PSObject[]]$pid = ''
$getUri1 = 'https://playbook2.com/data/project/folder/28220'
$projectIds = wget $getUri1 -UseDefaultCredentials |
ConvertFrom-JSON | Select data | select -Expand data | select id
Write-Host $projectIds
#obtain all the project ids
ForEach-Object{
[PSObject[]]$pid += $projectIds.id
}
Write-Host $pid
$uri3 = "https://playbook2.com/data/project/export/projects-tasks?projectIds[]="
$getIds = [PSObject[]]$pid -join "&projectIds[]="
$getUri2 = $uri3 + $getIds
$of = "\\ant\dept\DCGSI\Extracts\Time_Tracking_Tasks.xlsx"
Write-Host $getUri2
#retrieve excel files of tasks from each sub-folder
wget $getUri2 -outfile $of -UseDefaultCredentials
}
This code is an adaptation of some other code that I wrote. The 5 other scripts work fine. The main difference is that the other code has to loop through multiple folders and gets the project IDs under each folder, but this code only has to go through a single folder. Now in the other code the $uri3, $getIds code works fine and I get an export. The problem I am seeing in this code is that it isn't joining the URL the way I expect.
https://playbook2.com/data/project/export/projects-tasks?projectIds[]=######&projectIds[]=####### is the expected and previously seen output to get all the project data i need.
The problem with the above script is that it is giving https://playbook2.com/data/project/export/projects-tasks?projectIds[]=&projectIds[]=######&projectIds[]=####### which is invalid.
is there a way that I can tell it to do just $pid for the first item in the object and then -join the "&projectIds[]=" on the next n until the end of the list? I tried
[PSObject[]]$pid | select -Skip 1 -join "&projectIds[]="
and
[PSObject[]]$pid | Select-Object -Skip 1 -join "&projectIds[]="
but that results in nothing being appended.
I found a couple of "mistakes" in your script.
First is that you are using the variable $pid which is an system default variable. You can check the system global variables by typing
Get-Variable
Secondly $pid is defined with an empty string. The correct way to initialize a PSObject is with $myVar = New-Object PSObject. Replace [PSObject[]]$pid = '' with $myProjectIds = New-Object PSObject
For readability I took the liberty to rewrite your script.
function Get-Data(){
$GetProjectsUri = 'https://playbook2.com/data/project/folder/28220'
$ExportProjectsUri = 'https://playbook2.com/data/project/export/projects-tasks?'
$ExportFilePath = "\\ant\dept\DCGSI\Extracts\Time_Tracking_Tasks.xlsx"
$GetProjectsJson = Invoke-WebRequest -Uri $GetProjectsUri -UseDefaultCredentials
Write-Output $GetProjectsJson
$Projects = ConvertFrom-JSON -InputObject $GetProjectsJson
Write-Output $Projects
foreach ($Project in $Projects) {
$ProjectId = $Project.data.id
# Check if ProjectId exists
if ($ProjectId) {
$ExportProjectsUri = $ExportProjectsUri + 'projectIds[]=' + $ProjectId
}
}
Write-Output $ExportProjectsUri
Invoke-WebRequest Invoke-WebRequest -Uri $ExportProjectsUri -outfile $ExportFilePath -UseDefaultCredentials
}
Cheers
Glenn