I have a PowerShell script to verify the existence of many (half a million) files in a web server. Some of these files are large (gigabytes) and I don't wan't to transfer them, so I've been using the HTTP HEAD method.
The problem is, today I found that the HTTP status code returned from the HEAD method may be different from the status returned by GET (which is always the correct one):
Using HEAD:
> Invoke-WebRequest -Method head http://example.com/82833749.mbtiles
StatusCode : 200
StatusDescription : OK
Content : {}
RawContent : HTTP/1.1 200 OK
Content-MD5: LwIBDR5QRptNAE2Hdpu+aw==
x-ms-request-id: 774da565-101e-0053-7903-38e136000000
x-ms-version: 2009-09-19
x-ms-lease-status: unlocked
x-ms-blob-type: BlockBlob
Connect...
Headers : {[Content-MD5, LwIBDR5QRptNAE2Hdpu+aw==], [x-ms-request-id,
774da565-101e-0053-7903-38e136000000], [x-ms-version, 2009-09-19], [x-ms-lease-status,
unlocked]...}
RawContentLength : 0
With GET:
> Invoke-WebRequest http://example.com/82833749.mbtiles
Invoke-WebRequest : BlobNotFoundThe specified blob does not exist. RequestId:dede785b-b01e-0091-4b08-38a7b0000000
Time:2022-03-15T01:02:36.7883837Z
No linha:1 caractere:1
+ Invoke-WebRequest http://example.com/82833749.mbtiles
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
> $Error[0].Exception.Response.StatusCode.value__
404
Right after running the GET request, the HEAD request will return the correct status code, making it more confusing to understand what is going on.
Related
I'm writing a script that will download Google sheets using an Oauth access token. The script works just fine, but I'm having issues refreshing my access token.
Every guide I have found online shows me some iteration of the following:
$refreshTokenParams = #{
client_id=$clientId;
client_secret=$secret;
refresh_token=$refreshToken;
grant_type='refresh_token';
}
$refreshedToken = Invoke-WebRequest -Uri "https://accounts.google.com/o/oauth2/token" -Method POST -Body $refreshTokenParams
$accesstoken = $refreshedToken.access_token
When I run this script it returns the following:
StatusCode : 200
StatusDescription : OK
Content : <!doctype html><html lang="en" dir="ltr"><head><base href="https://accounts.google.com/"><script data-id="_gd" nonce="<Hidden just in case>">window.WIZ_global_data =
{"Mo6CHc":-<Hidden just in case>,"O...
RawContent : HTTP/1.1 200 OK
X-Frame-Options: DENY
Vary: Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site
google-accounts-embedded: 1
Pragma: no-cache
Transfer-Encoding: chunked
Strict-Transport-Security: max-...
Forms : {}
Headers : {[X-Frame-Options, DENY], [Vary, Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site], [google-accounts-embedded, 1], [Pragma, no-cache]...}
Images : {}
InputFields : {}
Links : {#{innerHTML=Learn more; innerText=Learn more; outerHTML=Learn more;
outerText=Learn more; tagName=A; href=https://developers.google.com/identity/protocols/oauth2; target=_blank; jsname=erTfTe}, #{innerHTML=Help; innerText=Help; outerHTML=<A
href="https://support.google.com/accounts?hl=en" target=_blank>Help</A>; outerText=Help; tagName=A; href=https://support.google.com/accounts?hl=en; target=_blank},
#{innerHTML=Privacy; innerText=Privacy; outerHTML=<A href="https://accounts.google.com/TOS?loc=US&hl=en&privacy=true" target=_blank>Privacy</A>; outerText=Privacy;
tagName=A; href=https://accounts.google.com/TOS?loc=US&hl=en&privacy=true; target=_blank}, #{innerHTML=Terms; innerText=Terms; outerHTML=<A
href="https://accounts.google.com/TOS?loc=US&hl=en" target=_blank>Terms</A>; outerText=Terms; tagName=A; href=https://accounts.google.com/TOS?loc=US&hl=en; target=_blank}}
ParsedHtml : System.__ComObject
RawContentLength : 1759969
When I save this output to an HTML file, I get this
Error 400: invalid_request
The error says "Required parameter is missing: response_type"
This Google doc mentions response_type='code' and I've added that to my array and that had no impact.
I feel like this section in the guide SHOULD work, but it doesnt. Unless maybe I'm implementing it wrong?
I have tried using "Invoke-restmethod" while specifying the content type to json/application, I've used alternative URIs and I've quadruple checked my client ID and password. I have no idea what I'm doing wrong.
If anyone has experience with refreshing Oauth access tokens using Powershell I would really appreciate your help.
Thanks in advance
I wrote this a while ago for gmail api GmailSendMail.psi
The issue is how you are sending the post body. It needs to be in the query parameters format.
function RefreshAccessToken([string]$clientId,[string]$secret, [string]$refreshToken){
$data = "client_id=$clientId&client_secret=$secret&refresh_token=$refreshToken&grant_type=refresh_token"
try {
$response = Invoke-RestMethod -Uri https://www.googleapis.com/oauth2/v4/token -Method POST -Body $data
return $response.access_token;
} catch {
# Dig into the exception to get the Response details.
# Note that value__ is not a typo.
Write-Host "StatusCode:" $_.Exception.Response.StatusCode.value__
Write-Host "StatusDescription:" $_.Exception.Response.StatusDescription
}
}
Let me know if you have any issues i will see if i cant update it for you.
Want to initiate a VSTS git repo import from project A to Project B. Following script is written to createthe import request via REST API. It is giving bad request 400 as described below. Any clues?
param(
<parameter(mandatory=$true)>
[string] $token,
<parameter(mandatory=$true)>
[string] $collectionUri,
<parameter(mandatory=$true)>
[string] $TargetTeamProject,
<parameter(mandatory=$true)>
[string] $TargetGitRepoName,
<parameter(mandatory=$true)>
[string] $SourceGitRepoUrl
)
$User=""
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f $User,$token)));
$header = #{Authorization=("Basic {0}" -f $base64AuthInfo)};
$Uri = $collectionUri + $TargetTeamProject + '/_apis/git/repositories/' + $TargetGitRepoName + '/importRequests?api-version=4.1-preview.1'
$ImportRequestData = '{"parameters": {"gitSource": {"url": "' + $SourceGitRepoUrl + '"}}}'
write-host 'calling:' $Uri
write-host 'with data:' $ImportRequestData
$ImportResponse = Invoke-RestMethod -Method Post -ContentType application/json -Uri $Uri -Body $ImportRequestData -Headers $header
$ImportResponse
Called it with
.\CreateImportRequest.ps1 -token '*************************************' -collectionUri 'https://myvsts.visualstudio.com/DefaultCollection/' -TargetTeamProject 'HasiTempJava' -TargetGitRepoName 'impfromhasiJava' -SourceGitRepoUrl 'https://myvsts.visualstudio.com/DefaultCollection/_git/HasiJava'
When passed existing empty repo it throws bad request exception
Invoke-RestMethod : The remote server returned an error: (400) Bad Request.
At C:\Users\chamindac\Desktop\CreateImportRequest.ps1:33 char:19
+ ... tResponse = Invoke-RestMethod -Method Post -ContentType application/j ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand</parameter(mandatory=$true)></parameter(mandatory=$true)></parameter(mandatory=$true)></parameter(mandatory=$true)></parameter(mandatory=$true)>
When a new repo name used it is giving below error which is correct I believe since url refers to a non existing repo
<parameter(mandatory=$true)><parameter(mandatory=$true)><parameter(mandatory=$true)><parameter(mandatory=$true)><parameter(mandatory=$true) class="">Invoke-RestMethod : {"$id":"1","innerException":null,"message":"TF401019: The Git repository with name or identifier impfromhasiJavaNew does not exist or you do not have permissions for the operation you are
attempting.","typeName":"Microsoft.TeamFoundation.Git.Server.GitRepositoryNotFoundException, Microsoft.TeamFoundation.Git.Server","typeKey":"GitRepositoryNotFoundException","errorCode":0,"eventId":3000}
At C:\Users\chamindac\Desktop\CreateImportRequest.ps1:33 char:19
+ ... tResponse = Invoke-RestMethod -Method Post -ContentType application/j ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
+ FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand </parameter(mandatory=$true)></parameter(mandatory=$true)></parameter(mandatory=$true)></parameter(mandatory=$true)></parameter(mandatory=$true)>
As per documentation (https://learn.microsoft.com/en-us/rest/api/vsts/git/import%20requests/create) only required parameter is the source git repo url. Is there any additional parameter requirements?
Post url used
https://myvsts.visualstudio.com/DefaultCollection/HasiTempJava/_apis/git/repositories/impfromhasiJava/importRequests?api-version=4.1-preview.1
Request Body
{"parameters": {"gitSource": {"url": "https://myvsts.visualstudio.com/DefaultCollection/_git/HasiJava"}}}
You need to add the source repo (from project A) as an External Git Endpoint for the target project (project B), and then provide the argument serviceEndpointId for create import requests REST API.
Detail steps ad below:
Create External Git Endpoint in target project
In project B -> Services Hub (https://account.visualstudio.com/projectB/_admin/_services) -> New Service Endpoint -> External Git -> add source repo from project A as the server URL (https://account.visualstudio.com/projectA/_git/sourcerepo) -> OK.
Get the endpoint id
Use the REST API as below:
GET https://marinaliu.visualstudio.com/GitTest/_apis/serviceendpoint/endpoints/?api-version=4.1-preview.1
Then get the endpoint id from the response (assume it's c534772b-bf52-442f-abd0-544d6bf76ed9).
import git repo to the target porject
Then import the source repo from project A to project B:
POST https://account.visualstudio.com/projectB/_apis/git/repositories/targetrepo/importRequests?api-version=4.1-preview.1
application/json:
{
"parameters": {
"gitSource": {
"url": "https://marinaliu.visualstudio.com/projectA/_git/sourcerepo"
},
"serviceEndpointId": "c534772b-bf52-442f-abd0-544d6bf76ed9"
}
}
More details, you can refer the blog Import a Git Project with REST API between VSTS Team Projects.
I want to make a backup script for ZyWall USG 20.
I used this Powershell code to get a list of the content in the directory:
[System.Net.FtpWebRequest]$ftp = [System.Net.WebRequest]::Create("ftp://*server*/*path*")
$ftp.Credentials = New-Object System.Net.NetworkCredential("*user*","*password*")
$ftp.Method = [System.Net.WebRequestMethods+FTP]::ListDirectory
$response = $ftp.getresponse()
echo $response
I expected to get a list of all the files..instead I get this:
PS Z:\> ...\list-files-test.ps1
ContentLength : -1
Headers : {}
SupportsHeaders : True
ResponseUri : ftp://*server*/*path*/
StatusCode : OpeningData
StatusDescription : 150 Opening BINARY mode data connection for file list
LastModified : 01.01.0001 00:00:00
BannerMessage : 220 FTP Server (ZyWALL USG 20) [::ffff:*server*]
WelcomeMessage : 230 User *user* logged in
ExitMessage :
IsFromCache : False
IsMutuallyAuthenticated : False
ContentType :
PS Z:\>
Why don' I get a list of files?
I checked it with filezilla - the folder is not empty.
And I can download specific files using [System.Net.WebRequestMethods+Ftp]::DownloadFile.
WebRequest.GetResponse() makes the remote call and gets the response details, but there's no content within the response itself.
Instead, you need to continue on and read the response stream out of the response with WebResponse.GetResponseStream().
I've got a curl command that works great and I'm trying to turn it into a PowerShell Invoke-WebRequest call.
Here's the working curl with the multipart POST data:
curl -X POST "http://domain.com/admin/batchOrder/processBatch"
-F "ordersFile=#c:\temp\Sample_Auto_Order_Input_File.xml"
-F "psid=3002010250764"
-F "recipients[0]=astraljack#example.com"
And here's where I'm trying to do it via Invoke-WebRequest. Based on what I've read, I need to create a hash table of name-value pairs. I feel like I should be creating the hash table just like I do the multipart POST data above.
$form.clear()
$form.Add("ordersFile", "c:\temp\Sample_Auto_Order_Input_File.xml")
$form.Add("psid", "3002010250764")
$form.Add("recipients[0]", "astraljack#example.com")
$form
Then I can call Invoke-WebRequest.
Invoke-WebRequest http://domain.com/admin/batchOrder/processBatch -Body $form -Method Post
This fails with a 400 Bad Request. To be clear, I've never gotten this Invoke-WebRequest to work so I could be way off base here. I'm not even sure where I can look for a better error msg.
If it helps, the contents of the exception are
IsMutuallyAuthenticated : False
Cookies : {}
Headers : {Connection, Transfer-Encoding, Content-Type, Date...}
SupportsHeaders : True
ContentLength : -1
ContentEncoding :
ContentType : application/json;charset=UTF-8
CharacterSet : UTF-8
Server : Apache
LastModified : 5/12/2014 10:23:19 AM
StatusCode : BadRequest
StatusDescription : Bad Request
ProtocolVersion : 1.1
ResponseUri : http://domain.com/admin/batchOrder/processBatch
Method : POST
IsFromCache : False
The best way to solve the problem is to go download Fiddler and then capture the CURL request. Inspect its RAW request. Then do the same for the Invoke-WebRequest command. See what is different and that should tell you what you need to tweak with Invoke-WebRequest.
The #filepath syntax in curl actually reads the contents of the file into the parameter, but in your PowerShell version you are just setting the value of ordersFile to the filename. Try:
$form.clear()
$form.Add("ordersFile", (get-content "c:\temp\Sample_Auto_Order_Input_File.xml"))
$form.Add("psid", "3002010250764")
$form.Add("recipients[0]", "astraljack#example.com")
$form
Also, you may need to add -ContentType "application/x-www-form-urlencoded" to your Invoke-WebRequest call.
`
I know this is an old question, but I figured since it was still unanswered, I'd throw in my two cents. Here's how I would do this:
$form = #{
'ordersFile' = 'c:\temp\Sample_Auto_Order_Input_File.xml'
'psid' = '3002010250764'
'recipients[0]' = 'astraljack#example.com'
}
Invoke-WebRequest -Uri "http://domain.com/admin/batchOrder/processBatch" -Body $form -Method Post
Im building a powershell script to execute commands on a REST-like server, I am using powershell to POST XML data to the webservice. Everything is working perfectly except when the "application" produces an error. The application replies back with details in xml and also gives me a Web Request Error (in this case 400) which is when i have problems. Whenever i get a Web Request Error that is not 200 the commands throw an error which screws up how i handle the error.
This is an example of the response im receiving (using fiddler):
HTTP/1.1 400 Bad Request
Date: Thu, 29 Nov 2012 04:29:48 GMT
Content-Type: text/xml;charset=ISO-8859-1
Connection: close
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Status>
<operation>Add Network</operation>
<result>ERROR</result>
<resultCode>REASON_210</resultCode>
</Status>
This is the error i receive in powershell:
Exception calling "UploadData" with "3" argument(s): "The remote server returned an error: (400) Bad Request."
At C:\Scripting\test.ps1:47 char:34
+ $response = $query.UploadData <<<< ($url,"POST",$byteArray)
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : DotNetMethodException
Exception calling "GetString" with "1" argument(s): "Array cannot be null.
Parameter name: bytes"
At C:\Scripting\test.ps1:48 char:62
+ $stringresponse = [System.Text.Encoding]::ASCII.GetString <<<< ($response)
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : DotNetMethodException
I dug further into the error and this is as close as i could get to retrieving the content of the response (the XML).
I thought I might be able to pull it out from this..
$error[1].exception.InnerException.Data
But it is empty :(
PS C:\Scripting> $error[1].exception.InnerException | fl -force
Status : ProtocolError
Response : System.Net.HttpWebResponse
Message : The remote server returned an error: (400) Bad Request.
Data : {}
InnerException :
TargetSite : Byte[] UploadDataInternal(System.Uri, System.String, Byte[], System.Net.WebRequest ByRef)
StackTrace : at System.Net.WebClient.UploadDataInternal(Uri address, String method, Byte[] data, WebRequest& request)
at System.Net.WebClient.UploadData(Uri address, String method, Byte[] data)
at System.Net.WebClient.UploadData(String address, String method, Byte[] data)
at UploadData(Object , Object[] )
at System.Management.Automation.MethodInformation.Invoke(Object target, Object[] arguments)
at System.Management.Automation.DotNetAdapter.AuxiliaryMethodInvoke(Object target, Object[] arguments, MethodInformation methodInformation, Object[]
originalArguments)
HelpLink :
Source : System
PS C:\Scripting> $error[1].exception.InnerException.response | fl
IsMutuallyAuthenticated : False
Cookies : {}
Headers : {Connection, Transfer-Encoding, Content-Type, Date}
ContentLength : -1
ContentEncoding :
ContentType : text/xml;charset=ISO-8859-1
CharacterSet : ISO-8859-1
Server :
LastModified : 29/11/12 4:04:19 PM
StatusCode : BadRequest
StatusDescription : Bad Request
ProtocolVersion : 1.1
ResponseUri : https://api-au.dimensiondata.com/oec/0.9/cdefb891-1552-4008-8638-7393eda8d7e1/network
Method : POST
IsFromCache : False
Here is my powershell code that I am using:
function send_web ($url,$data) {
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
$query = new-object system.net.WebClient
$auth = 'Basic ' + [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($username+":"+$password ))
$query.Headers.Add('Authorization', $auth )
[byte[]]$byteArray = [System.Text.Encoding]::ASCII.GetBytes($data)
$query.Headers.Add("ContentType","application/xml")
$response = $query.UploadData($url,"POST",$byteArray)
$stringresponse = [System.Text.Encoding]::ASCII.GetString($response)
return $stringresponse
}
This seems like it shouldn't be a hard thing to do any help would be much appreciated :)
Richard