I want to get the content of a web page and when I use
$web = Invoke-RestMethod -Uri "https://inkscape.org/"
I will get the content but when I use
$web = Invoke-WebRequest -Uri "https://inkscape.org/"
I won't get anything why it happens?? and what is the difference exactly ??
Simply put, for plain-text or HTML response bodies, the relationship between the (older) Invoke-WebRequest cmdlet and the Invoke-RestMethod cmdlet is as follows with the respect to the default GET method:
# -UseBasicParsing is only needed in *Windows PowerShell*.
(Invoke-WebRequest -UseBasicParsing -Uri "https://inkscape.org/").Content
is the same as:
Invoke-RestMethod -Uri "https://inkscape.org/"
That is:
Invoke-WebRequest returns a response object whose .Content property contains the body of the response, always as text (except if you save the raw body to a file with -OutFile).
For HTML response bodies, Windows PowerShell attempts to also parse the HTML text into an HTML DOM, surfaced via the .ParsedHTML property, using the obsolete Internet Explorer. -UseBasicParsing suppresses this. This switch has no effect in PowerShell (Core) 7+, which fundamentally doesn't support parsing HTML, requiring third-party solutions (see this answer for an example) or - on Windows only - a COM-based solution (see this answer).
Invoke-RestMethod directly returns the response body (only).
Additionally, if the target site indicates that XML or JSON data is being returned, Invoke-RestMethod doesn't return the body as text, but automatically parses it into an [xml] instance / [System.Xml.XmlElement] instances (for RSS / Atom feeds) or a [pscustomobject] graph (ConvertFrom-Json is built in, so to speak).
Even in the absence of a known response data format, PowerShell tries to parse the response body, first as XML, then as JSON; if all attempts fail, plain text (a [string] instance) is returned.
Even for text/html responses an attempt is made to parse them as XML. That is, if a page happens to be valid XML (which is rare these days), you'll get an [xml] instance back; for instance, the very simple HTML5 page at https://httpbin.org/html happens to be valid XML (excluding the <!DOCTYPE html> declaration), whereas HTML5 pages in general are not. Thus, (Invoke-RestMethod https://httpbin.org/html).GetType().FullName returns System.Xml.XmlDocument, i.e. an [xml] instance.
Related
I am trying to extract a streaming response from a standard http GET call using PowerShell.
I tried to do something like using this Invoke-RestMethod method, such as:
$response = Invoke-RestMethod -Uri $uri -Method GET -Headers $headers -Body
and then tried doing various things like accessing the streaming response in this way:
$Stream.Write($response.Content)
or
$Stream.Write($response.RawContentStream)
but nothing works.
In the end, I found out the following command does both the call and the response along with allowing you to save the result to a file directly. This surprised me because the response is a multipart record that includes headers and other data, not just the chunked file in the stream. But the folks at MS apparently knew that 99% of the time with a chunked reply, the user only wants to save the streamed chunks of data into the file, without the other response header data.
$Response = Invoke-WebRequest #Params -Headers $headers -outfile $TempZipFileName
The key here is using the Invoke-WebRequest call with the -outfile option. This saved me a lot of headache and was super simple to use.
There's been a few similar posts on this but none quite the same (or they don't describe the issue clearly, if they are the same).
I've got a script which makes one API call using Invoke-RestMethod and then iterates over the result of that call making other calls with values from it. The value substitution etc is fine, and I can see (both by dumping the Uri out with Write-Host, and also from the API logs itself) that the calls are being made successfully.
If I do this:
$team = #{ "id" = 5 }
$response = Invoke-RestMethod -Uri https://theUri/$($team.id) -Headers $header -Method Get
then $response is populated as you'd expect and contains the json response from the API endpoint. Response time is a few hundred milliseconds.
If, on the other hand, I have an array of team objects and do this:
foreach ($team in $teams)
{
$response = Invoke-RestMethod -Uri https://theUri/$($team.id) -Headers $header -Method Get
Write-Host $response
}
then it iterates over the whole of the $teams array and outputs empty values on each of the Write-Host lines (the same occurs no matter what you do with it, I'm just using Write-Host as an example here; sending it to Select-Object etc shows it's still blank).
Is this deliberate? I assume no, because it's mad. I've got a temporary workaround by (yes, it's this awful...) writing out the results of the first Api call to a file, then iterating over that list calling a script that does the subsequent calls without the foreach loop in the script for each row. Is there a better way?
I've tried -DisableKeepAlive - no difference. I've tried setting the -TimeoutSec value to a small number, a big number, etc - no difference. I've tried -OutFile and that writes out a bunch of empty files.
It feels - without any evidence for this - that when called inside the loop it's waiting for the HTTP response (which is a 200) and then not downloading any of the response body.
This is Powershell 7.1.3, btw.
Cheers in advance!
I am trying to upload a file to a URL. I have tried both these approaches:
Invoke-RestMethod -Uri $uploadUrl -Method Put -Headers $uploadHdrs -InFile $uploadFilePath
Invoke-RestMethod -Uri $uploadUrl -Method Put -body $uploadFileBody -Headers $uploadHdrs
Error I am getting:
Invoke-RestMethod : You must write ContentLength bytes to the request stream
before calling [Begin]GetResponse.
If I add in the -TransferEncoding param, I get errors from the server saying unsupported.
What can I do to include the content length?
I think you'll need to use a different command, like Invoke-WebRequest or even better call the .NET WebClient.UploadFile() or .UploadData methods directly from PowerShell.
While REST methods might conceptually include uploading files, that doesn't mean that the Invoke-WebRequest command has been tested as supporting your scenario. I'd suggest going lower-level (like WebClient) because we know more scenarios have been tested (by the larger .NET team), and there are a wide variety of methods on WebClient for supporting specific scenarios.
One example you might find helpfull in how to invoke these methods from PowerShell is at https://social.technet.microsoft.com/Forums/windowsserver/en-US/0c268c7e-674c-49bc-9933-a87a95f8f44c/powershell-webclientuploadfile?forum=winserverpowershell
P.S. The message your are getting about the request stream is coming from an even lower level .NET API, but WebClient is simpler to use, should take care of setting ContentLength properly and hopefully is "just right" for your need.
I am using a PowerShell module to interact with a manufacturer's REST API. The module is written by the manufacturer and makes use of the Invoke-RestMethod built in function.
I noticed when running my program with few outputs turned on that I was getting a ton of lines appearing in the console. I had never seen that before, so i decided to dig in. I traced the issue down to a single line of code:
Invoke-RestMethod -Headers $session.DirectoryAuthHeader -Uri $uri -Method $Method -Body $Body
I further instrumented it bu outputting to the console all of the variables that are being sent to the function. This issue only seems to happen when Invoke-RestMethod is using the PUT method. GET and POST do not yield any new lines. I took a look at the transactions in Fiddler and the only thing that jumped out at me was the PUT method returned a blank body.
Any ideas? I'm hoping this is just an issue with the way Invoke-RestMethod is being called and not the method itself. Thanks.
I ended up solving the issue myself. I'm still not 100% why PowerShell is functioning this way, but basically the blank return value (because there is no body in the response) is getting echoed to the command line. I fixed the issue by checking if the body was empty, and if not returning null instead of an empty PSObject which I believe is what was happening.
$RESTResponse = Invoke-RestMethod -Headers $session.DirectoryAuthHeader -Uri $uri -Method $Method -Body $Body
if ($RESTResponse -ne "") {
return $RESTResponse
} else {
return
}
As I've done some more learning in PowerShell I found out what this is happening and wanted to share it for anyone else who comes across this problem.
In PowerShell there is a concept called the pipeline. Basically what happens is if cmdlets are called in succession using the pipeline character "|", the output from the preceding function becomes the input of the next function and so on. If there is output on the pipeline that isn't handled (e.g. putting it in a variable) it will eventually be echoed to the host.
I wrote a small PowerShell script to send a request to a server and get a simple XML result back.
PowerShell Script
$xml = "<?xml version='1.0' encoding='utf-8'?><Search><ID>278A87E1-1BC2-4E19-82E9-8BBE31D67D20</ID></Search>"
$response = Invoke-RestMethod -Method Post -Uri "http://localhost/search" -ContentType "application/xml" -Body $xml
That's it, really simple and there's no reason that I can see for it to be failing. I have also tried the script with Invoke-WebRequest and both fail. The error returned is Invoke-RestMethod : Value cannot be null. Parameter name: name. The strange thing is that when I monitor this with Wireshark, I see the connection, I see the POST and I see the result from the server and all looks perfectly good but the cmdlet is saying it failed (and yes, the return code is 200).
If I run the Invoke-WebRequest/Invoke-RestMethod with the -OutFile parameter, it runs fine with no errors and saves the result to the specified file; -OutVariable fails just in case you're wondering.
The result is an xml file, the headers specify that it is xml and the xml is properly formatted.
Result when successful
<?xml version="1.0" encoding="UTF-8" ?>
<Result version="1.0" xmlns="urn:xmlns-org">
<searchID>{278a87e1-1bc2-4e19-82e9-8bbe31d67d20}</searchID>
<responseStatus>true</responseStatus>
<responseStatusStrg>MORE</responseStatusStrg>
<numOfMatches>40</numOfMatches>
</Result>
Does anyone know why the Invoke-XXX cmdlets are returning an error and what I can do to fix it? Again, it works perfectly fine when I use the -OutFile parameter and even when it fails I can see a proper conversation between the script and the server in Wireshark.
Also, if I use -Verbose it tells me the following:
VERBOSE: POST http://localhost/search with -1-byte payload
VERBOSE: received X-byte response of content type application/xml; charset="UTF-8"
Where X-byte is the actual size of the response but it obviously differs with each response depending on the data sent to the server. I just find it odd that the cmdlet fails but says it received a response with data and that it sent a -1-byte payload.
I went ahead and looked into the Invoke-WebRequest cmdlet code and found out why it's failing with this particular error.
It's failing on a call to System.Globalization.EncodingTable.GetCodePageFromName. The encoding is passed to that function as a parameter and the encoding is retrieved from the the cmdlet through the Content-Type header. In the case of this server the Content-Type was sent back in the response as Content-Type: application/xml; charset="UTF-8".
The problem with this is that quotes aren't standard for wrapping the value in charset so the cmdlet parses it out as "UTF-8" instead of the valid UTF-8. The cmdlet passes "UTF-8" to the function and the function throws an exception stating that the provided encoding is invalid. This is fine and would make so much more sense if that is what was reported in the final exception but it's not.
The Invalid encoding exception is caught by the Microsoft.PowerShell.Commands.ContentHelper.GetEncodingOrDefault function and in the exception handler it calls GetEncoding again but with a null parameter which results in the final ArgumentNullException for parameter name.
Microsoft.PowerShell.Commands.ContentHelper.GetEncodingOrDefault
internal static Encoding GetEncodingOrDefault(string characterSet)
{
string name = string.IsNullOrEmpty(characterSet) ? "ISO-8859-1" : characterSet;
try
{
return Encoding.GetEncoding(name);
}
catch (ArgumentException ex)
{
return Encoding.GetEncoding((string) null);
}
}
The call to GetEncoding inside the catch statement triggers the following code inside GetCodePageFromName which is itself called from GetEncoding
if (name==null) {
throw new ArgumentNullException("name");
}
PowerShell is handling this properly since technically it is an invalid value but you'd think they would call Trim("\"") just to be safe.
Although this thread is years-old, I came across it in 2022 when I encountered the same error 'Invoke-RestMethod : Value cannot be null. Parameter name: name' accessing an API to create a user. In my case Content-Type was application/json (rather than application/xml in Vane's question above). The API POST required Basic Auth.
I UTF8 encoded the credentials prior to converting to Base64, and et voilà the error disappeared. Given that the Content-Type investigation by Vane above tracked a return of charset="UTF-8", this UTF8 encoding and the disappearance of the error may be related, although this is just surmise on my part.
$username = "your_username"
$password = "your_password"
$credential = "${username}:${password}"
$credentialBytes = [System.Text.Encoding]::UTF8.GetBytes($credential)
$encodedCredential = [System.Convert]::ToBase64String($credentialBytes)
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Authorization", "Basic $encodedCredential")
$headers.Add("Content-Type", "application/json")