I am attempting to validate URLs prior to downloading those files, and I want to differentiate between things like a path that requires credentials vs a bad path.
I have this at the moment
function Get-UrlStatusCode([string] $Url) {
try {
return (Invoke-WebRequest -Uri $Url -UseBasicParsing -DisableKeepAlive -Method:head).StatusCode
} catch [Net.WebException] {
return "$([Int]$_.Exception.Response.StatusCode) $($_.Exception.Response.statusDescription)"
}
}
(Get-UrlStatusCode 'http://AWSBUCKETPATH/Test_Public.xml')
(Get-UrlStatusCode 'http://AWSBUCKETPATH/Test_Private.xml')
(Get-UrlStatusCode 'http://AWSBUCKETPATH/Test_Missing.xml')
For the public path I am getting the expected StatusCode of 200, but both the private and the missing examples are returning 403 Forbidden. Is this a failure in my code, or something that needs to be configured at AWS to provide a 404 Not Found for the bad path, or is this just not something that is possible despite the existence of 404 as a code?
Related
I'm writing a script that will download Google sheets using an Oauth access token. The script works just fine, but I'm having issues refreshing my access token.
Every guide I have found online shows me some iteration of the following:
$refreshTokenParams = #{
client_id=$clientId;
client_secret=$secret;
refresh_token=$refreshToken;
grant_type='refresh_token';
}
$refreshedToken = Invoke-WebRequest -Uri "https://accounts.google.com/o/oauth2/token" -Method POST -Body $refreshTokenParams
$accesstoken = $refreshedToken.access_token
When I run this script it returns the following:
StatusCode : 200
StatusDescription : OK
Content : <!doctype html><html lang="en" dir="ltr"><head><base href="https://accounts.google.com/"><script data-id="_gd" nonce="<Hidden just in case>">window.WIZ_global_data =
{"Mo6CHc":-<Hidden just in case>,"O...
RawContent : HTTP/1.1 200 OK
X-Frame-Options: DENY
Vary: Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site
google-accounts-embedded: 1
Pragma: no-cache
Transfer-Encoding: chunked
Strict-Transport-Security: max-...
Forms : {}
Headers : {[X-Frame-Options, DENY], [Vary, Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site], [google-accounts-embedded, 1], [Pragma, no-cache]...}
Images : {}
InputFields : {}
Links : {#{innerHTML=Learn more; innerText=Learn more; outerHTML=Learn more;
outerText=Learn more; tagName=A; href=https://developers.google.com/identity/protocols/oauth2; target=_blank; jsname=erTfTe}, #{innerHTML=Help; innerText=Help; outerHTML=<A
href="https://support.google.com/accounts?hl=en" target=_blank>Help</A>; outerText=Help; tagName=A; href=https://support.google.com/accounts?hl=en; target=_blank},
#{innerHTML=Privacy; innerText=Privacy; outerHTML=<A href="https://accounts.google.com/TOS?loc=US&hl=en&privacy=true" target=_blank>Privacy</A>; outerText=Privacy;
tagName=A; href=https://accounts.google.com/TOS?loc=US&hl=en&privacy=true; target=_blank}, #{innerHTML=Terms; innerText=Terms; outerHTML=<A
href="https://accounts.google.com/TOS?loc=US&hl=en" target=_blank>Terms</A>; outerText=Terms; tagName=A; href=https://accounts.google.com/TOS?loc=US&hl=en; target=_blank}}
ParsedHtml : System.__ComObject
RawContentLength : 1759969
When I save this output to an HTML file, I get this
Error 400: invalid_request
The error says "Required parameter is missing: response_type"
This Google doc mentions response_type='code' and I've added that to my array and that had no impact.
I feel like this section in the guide SHOULD work, but it doesnt. Unless maybe I'm implementing it wrong?
I have tried using "Invoke-restmethod" while specifying the content type to json/application, I've used alternative URIs and I've quadruple checked my client ID and password. I have no idea what I'm doing wrong.
If anyone has experience with refreshing Oauth access tokens using Powershell I would really appreciate your help.
Thanks in advance
I wrote this a while ago for gmail api GmailSendMail.psi
The issue is how you are sending the post body. It needs to be in the query parameters format.
function RefreshAccessToken([string]$clientId,[string]$secret, [string]$refreshToken){
$data = "client_id=$clientId&client_secret=$secret&refresh_token=$refreshToken&grant_type=refresh_token"
try {
$response = Invoke-RestMethod -Uri https://www.googleapis.com/oauth2/v4/token -Method POST -Body $data
return $response.access_token;
} catch {
# Dig into the exception to get the Response details.
# Note that value__ is not a typo.
Write-Host "StatusCode:" $_.Exception.Response.StatusCode.value__
Write-Host "StatusDescription:" $_.Exception.Response.StatusDescription
}
}
Let me know if you have any issues i will see if i cant update it for you.
I'm not very savvy with web API calls, but I've been using the following powershell code (this site in this example is one I found that has some public data... my site is internal and requires I pass the credential, which has been working for me without issue):
If(-not (Get-InstalledModule -Name 'ConfluencePS')){Install-Module ConfluencePS}
Import-Module ConfluencePS
Set-ConfluenceInfo -BaseUri "https://wiki.opnfv.org"
$space = Get-confluencepage -Spacekey ds
ForEach($item in $space)
{
$splatParams = #{
Uri = "https://wiki.opnfv.org/rest/api/content/$($item.ID)/restriction"
ContentType = 'application/json'
method = 'GET'
}
#reference https://developer.atlassian.com/cloud/confluence/rest/#api-api-content-id-restriction-get
Invoke-RestMethod #splatParams
}
The documentation for the ConfluencePS shows that restrictions is still an open feature request but I need to get this working for a project.
I put a breakpoint in on line 982 from ConfluencePS.psm1 and was able to see the various calls and how the params are structured but when I try to mimic it (and change the URI based on the confluence documentation) I get an error "HTTP error 405 - MethodNotAllowed". Anyone have suggestions on how I can get this working? I'm trying to return back the permissions applied for all pages in a specific space.
Get Restrictions by Content ID
As you found out by yourself, it is required to add "byOperation".
I was able to get the restrictions of a specific page with the following code:
# for testing purposes ONLY, I've specified the URL and ID
$wikiUrl = "https://wiki.opnfv.org"
$itemId = "6820746"
$splatParams = #{
Uri = "$wikiUrl/rest/api/content/$itemId/restriction/byOperation"
ContentType = 'application/json'
method = 'GET'
}
$result = Invoke-RestMethod #splatParams
Tested on version 6.0.4 and 6.15.9
Filter by user name
If you like to filter the result by a specific username, you can use the following URI:
"$wikiUrl/rest/api/content/$itemId/restriction/byOperation/.../user?userName=".
Bt, there's an open bug on this way of action:
restriction returns ambiguous responses
I'm using a standard [System.Net.WebRequest] class to return the HTTP response code of a given URL.
The URL points to an internal web application server, which returns a "401 Unauthorised" error. This is actually OK as the service account running the script doesn't have a valid account to the application. However, I am more interested that the website is responding. However, I assumed that this is a HTTP Response in itself so I could manage this, but instead it returned as a null value.
$HTTP_Request = [System.Net.WebRequest]::Create('http://google.com')
$HTTP_Response = $HTTP_Request.GetResponse()
$HTTP_Status = [int]$HTTP_Response.StatusCode
Exception calling "GetResponse" with "0" argument(s): "The remote
server returned an error: (407) Proxy Authentication Required."
(I'm using Google in this example, which our servers are blocked from accessing external sites).
So I can't get as far as $HTTP_Status = [int]$HTTP_Response.StatusCode in the code because it won't accept 400 errors as a code.
How can I accept a 401 (or 407 in this example) in the query?
Thanks
Got it!
try{
$request = $null
$request = Invoke-WebRequest -Uri "<URL>"
}
catch
{
$request = $_.Exception.Response
}
$StatusCode = [int] $request.StatusCode;
$StatusDescription = $request.StatusDescription;
You could've done:
$HTTP_Request = [System.Net.WebRequest]::Create('http://google.com')
$HTTP_Request.Method = "GET"
$HTTP_Request.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
[System.Net.HttpWebResponse]$HTTP_Response = $HTTP_Request.GetResponse()
Try {
$HTTP_Status = [int]$HTTP_Response.StatusCode
}
Catch {
#handle the error if you like, or not...
}
If ($HTTP_Status -eq 200) {
Write-Host "Good Response"
} Else {
Write-Host "Site Down or Access Denied"
}
# If you got a 404 Not Found, the response will be null so can't be closed
If ($HTTP_Response -eq $null) { } Else { $HTTP_Response.Close() }
You were missing the authentication piece:
$HTTP_Request.Credentials = [System.Net.CredentialCache]::DefaultNetworkCredentials
I re-read your post and because your service account did not have access to the URL you are hitting (which was not actually google.com... you should've put myserver.com...grr), you would never actually get a 200, but always will get an exception. This is a bad practice, because instead of looking for the 200, you would have to always look for the 401 or 407 Unauthorized exception status code specifically, and if the code/response changed to something else, only then is it considered a failure - but technically it always had been a failure, because it never reached the site! It masks potential issues if you intend to go on deliberately using a site your service account doesn't have access to reach. And if your service account was ever granted access, your code would have to be updated.
Having almost mastered using Powershell Invoke-WebRequest I'm now embarking on trying to get control of exception errors
In this first piece of code I've got to handle the exceptions returned from the web service ok. You can see a 400 code is returned along with a Message description and a detailed message.
Try {
$what = Invoke-WebRequest -Uri $GOODURL -Method Post -Body $RequestBody -ContentType $ContentType
}
catch [System.Net.WebException] {
$Request = $_.Exception
Write-host "Exception caught: $Request"
$crap = ($_.Exception.Response.StatusCode.value__ ).ToString().Trim();
Write-Output $crap;
$crapMessage = ($_.Exception.Message).ToString().Trim();
Write-Output $crapMessage;
$crapdescription = ($_.ErrorDetails.Message).ToString().Trim();
Write-Output $crapdescription;
}
Output returned. Nice!
Exception caught: System.Net.WebException: The remote server returned an error: (400) Bad Request.
at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.GetResponse(WebRequest request)
at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()
400
The remote server returned an error: (400) Bad Request.
Modus.queueFailed Could not queue message http://schemas.xmlsoap.org/soap/actor/next
That's very cool, however as part of my testing I wanted simulate connection errors, so I fed the script an invalid URI and immediately my error handling in the above code crashed.
Specifically, the invalid URI fell over on the "$_.Exception.Response.StatusCode.value__" and "$_.ErrorDetails.Message" presumably because they don't exist in the System.Net.WebException object returned
So, I took them out as a test and sure enough it works with the invalid URL as expected as below
Try {
$what = Invoke-WebRequest -Uri $BADURL -Method Post -Body $RequestBody -ContentType $ContentType
}
catch [System.Net.WebException] {
$Request = $_.Exception
Write-host "Exception caught: $Request"
$crapMessage = ($_.Exception.Message).ToString().Trim();
Write-Output $crapMessage;
}
And I get this (also nice)
Exception caught: System.Net.WebException: The remote name could not be resolved: 'gateway.zzzzsonicmobile.com'
at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
at System.Net.HttpWebRequest.GetRequestStream()
at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.SetRequestContent(WebRequest request, String content)
at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()
at System.Management.Automation.CommandProcessor.ProcessRecord()
The remote name could not be resolved: 'gateway.zzzzsonybile.com'
Trouble is, the first example gets everything I want from the exception object but won't work with an invalid URL. The second example handles the invalid URL (connection errors), but I can't figure out how to get hold of the 400 StatusCode and the ErrorDetails.Message
I would really like to do that, if at all possible.
Is there a way of setting this up to handle all exception handling..?
Any help, much appreciated..!
I think you're probably only seeing an issue because you're trying to do the .ToString().Trim() methods on those properties when they don't exist. I'm not sure doing those methods even adds a lot of value, so I'd be tempted to just remove them.
Alternatively, you could put If blocks around them so that they are only output if they have value:
If ($_.Exception.Response.StatusCode.value__) {
$crap = ($_.Exception.Response.StatusCode.value__ ).ToString().Trim();
Write-Output $crap;
}
If ($_.Exception.Message) {
$crapMessage = ($_.Exception.Message).ToString().Trim();
Write-Output $crapMessage;
}
I have created a PowerShell-based build task for Visual Studio Team Services (formerly Visual Studio Online). I have implemented the majority of the functionality I need, but for the last bit of functionality I need to be able to persist a small amount of data between builds.
The ExtensionDataService seems like exactly what I want (in particular, the setValue and getValue methods), but the documentation and examples I have found are for node.js-based build tasks:
VSS.getService(VSS.ServiceIds.ExtensionData).then(function(dataService) {
// Set a user-scoped preference
dataService.setValue("pref1", 12345, {scopeType: "User"}).then(function(value) {
console.log("User preference value is " + value);
});
The previous page also has a partial example of calling the REST API, but I have gotten 404 errors when trying to use it to either save or retrieve values:
GET _apis/ExtensionManagement/InstalledExtensions/{publisherName}/{extensionName}/Data/Scopes/User/Me/Collections/%24settings/Documents
{
"id": "myKey",
"__etag": -1,
"value": "myValue"
}
Can PowerShell be used to access the ExtensionDataService, either by using a library or by calling the REST API directly?
You can call REST API through PowerShell.
Set value (Put request):
https://[vsts name].extmgmt.visualstudio.com/_apis/ExtensionManagement/InstalledExtensions/{publisherName}/{extension id}/Data/Scopes/User/Me/Collections/%24settings/Documents?api-version=3.1-preview.1
Body (Content-Type:application/json)
{
"id": "myKey",
"__etag": -1,
"value": "myValue"
}
Get value (Get request):
https://[vsts name].extmgmt.visualstudio.com/_apis/ExtensionManagement/InstalledExtensions/{publisherName}/{extension id}/Data/Scopes/User/Me/Collections/%24settings/Documents/mykey?api-version=3.1-preview.1
The publisher name and extension id could be get in package json file (e.g. vss-extension.json)
Regarding call REST API through PowerShell, you can refer to this article: Calling VSTS APIs with PowerShell
Simple sample to call REST API:
Param(
[string]$vstsAccount = "<VSTS-ACCOUNT-NAME>",
[string]$projectName = "<PROJECT-NAME>",
[string]$buildNumber = "<BUILD-NUMBER>",
[string]$keepForever = "true",
[string]$user = "",
[string]$token = "<PERSONAL-ACCESS-TOKEN>"
)
# Base64-encodes the Personal Access Token (PAT) appropriately
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $user,$token)))
$uri = "https://$($vstsAccount).visualstudio.com/DefaultCollection/$($projectName)/_apis/build/builds?api-version=2.0&buildNumber=$($buildNumber)"
$result = Invoke-RestMethod -Uri $uri -Method Get -ContentType "application/json" -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)}
PowerShell script to get the base URL:
Function GetURL{
param([string]$url)
$regex=New-Object System.Text.RegularExpressions.Regex("https:\/\/(.*).visualstudio.com")
$match=$regex.Match($url)
if($match.Success)
{
$vstsAccount=$match.Groups[1]
$resultURL="https://$vstsAccount.extmgmt.visualstudio.com"
}
}
GetURL "https://codetiger.visualstudio.com/"