I'm trying to do something like this to parse a homepage with a login page but Invoke-WebRequest doesn't return anything.
The page I'm trying to access is https://www.suidoapp.waterworks.metro.tokyo.lg.jp/#/login and the code I'm running is this:
$TopURI = "https://www.suidoapp.waterworks.metro.tokyo.lg.jp/#/login"
$TopPage = Invoke-WebRequest -Method Get -Uri $TopURI -SessionVariable MySession -UseBasicParsing
When I look at the Content or RawContent of the $TopPage I can see that it just says "please enable JavaScript" (I've tried both with and without -UseBasicParsing). If I open the page in developer tool in my browser I can see that response for the initial document is the same:
But the interesting thing is that even though the initial page says "please enable JavaScript" the page actually loads:
Has anyone seen this before, where Invoke-WebRequest fails because the response is "please enable JavaScript" yet the page should actually be able to load? Is there another way for me to parse a homepage and send in login forms when Invoke-WebRequest fails like this?
I am having the same issue. The simple answer is: Invoke-WebRequest is not allowed to run javascript for fear of XSS attacks which makes total sense. In my case, I needed to run a Vue.js app via Task Scheduler (on Windows machine in conjunction with IIS). I eventually let PowserShell open a browser and finish the work then close it.
Start-Process -file iexplore -arg 'http://localhost:8080/ (or any URL)' -PassThru
sleep 10
(Get-Process -Name iexplore).Kill()
If you would like to run Firefox instead,
Start-Process -file 'C:\Program Files\Mozilla Firefox\firefox.exe (or your ff location)' -arg 'http://localhost:8080/ (or any URL)' -PassThru
sleep 10
(Get-Process -Name firefox).Kill()
Going back to your question, the loaded page is probably not functional if it invokes any javascript functions. If you talk about form login, you can find resources easily such as this: https://community.auth0.com/t/forms-login-via-curl-or-powershell/17456/3
I have a URL to a CSV file which, in a browser, I can download and open without issue.
I'm trying to download this file using PowerShell without success. I tried using Invoke-WebRequest, Start-BitsTransfer and using a webrequest object but no luck there.
Invoke-WebRequest comes with a parameter to store its result in a file: -OutFile
Invoke-WebRequest $myDownloadUrl -OutFile c:\file.ext
If you need authorization before you can send a request like this:
Invoke-WebRequest $myAuthUrl /* whatever is neccesary to login */ -SessionVariable MySession
Invoke-WebRequest $myDownloadUrl -WebSession $MySession
To determine the layout of the form where the login happens, you can use Invoke-WebRequests return object. It'll collect information about forms and fields on the HTML (might be Windows only). Mileage of logging in may vary with things like Two-Factor-Auth active or not. Probably you can create some secret link to your file which does not need Auth or possibly google allows you to create a private access token of some sort, which can be send aus Authorization-Header alongside your request.
TLDR answers*:
Method 1, by default synchronous**
Invoke-WebRequest $url -OutFile $path_to_file
(if you get error "...Could not create SSL/TLS secure channel." see Powershell Invoke-WebRequest Fails with SSL/TLS Secure Channel)
Method 2, by default synchronous**
(New-Object System.Net.WebClient).DownloadFile($url, $path_to_file)
Method 3, asynchronous and may be much slower than the other two but is very gentle on bandwidth usage (it uses the BITS service).
Import-Module BitsTransfer
Start-BitsTransfer -Source $url -Destination $path_to_file
Notes:
*: This answer is for those that google for "how to download a file with PowerShell".
**: Read the help pages if you want asynchronous downloading
For a while now I've been using a PS script to download PowerBI bi-monthly and using the BITS, it's been pretty solid and now so much stronger now since I removed the -Asynchronous at the end of the Start-BitsTransfer
$url = "https://download.microsoft.com/download/8/8/0/880BCA75-79DD-466A-927D-1ABF1F5454B0/PBIDesktopSetup.exe"
$output = "%RandomPath%\PowerBI Pro\PBIDesktopSetup.exe"
$start_time = Get-Date
Import-Module BitsTransfer
Start-BitsTransfer -Source $url -Destination $output
#Commented out below because it kept creating "Tmp files"
#Start-BitsTransfer -Source $url -Destination $output -Asynchronous
I am trying to download a google spreadsheet via an invoke-webrequestin powershell.
My link looks something like this and I can confirm that if I go to the link it prompts me to download the file I want...
$url = https://docs.google.com/spreadsheets/d/XXXXXXXXXXXXXXXX/export?format=csv
I have tried downloading the file via 2 ways:
Invoke-WebRequest $url -OutFile $saveLocation
(New-Object System.Net.WebClient).DownloadFile($url, $saveLocation)
Both of these just download the HTML for the page, and not the actual file.
The spreadsheet I am downloading is 100% public so it is not like I need to be logged into a google account to view it.
I made the test and I can quite safely say that your file is not public.
I have reproduced your issue before making my test sheet public, and I could not afterwards.
Tested with this :
$url = "https://docs.google.com/spreadsheets/d/1OOc5aiG3hh8mrGF4p7NJXKI8KqBSkS2ZjsBXtb4qpz8/export?format=csv"
Invoke-WebRequest $url -OutFile "C:\test.csv"
I have a sharepoint 2013 server, which I can log into using Windows Authentication via a web-browser. When I have logged on using my browser, if I then - using that same browser - browse to http://mysharepointserver/_api/web I get some useful XML.
However, I want to write a powershell script that uses this XML. To that end, following the suggestions from this page:
How to make an authenticated web request in Powershell?
I wrote the following code:
$PWord = ConvertTo-SecureString –String "MyAwesomePassword" –AsPlainText -Force
$creds = New-Object System.Net.NetworkCredential("MyUserName",$PWord,"TheDomain")
$webclient = New-Object System.Net.WebClient
$webclient.Credentials = $creds
$url = "http://mysharepointserver/_api/web"
$output = $webclient.DownloadString($url)
echo $output
However, when I run this code, I get the error message:
Exception calling "DownloadString" with "1" argument(s): "The remote server returned an error: (403) Forbidden."
Even though I can access the same URL using a web-browser, if I type in the (same) credentials using the Windows login dialog that pops up for authentication purposes.
Does anyone know if it is possible to do this using Powershell?
thanks heaps all
Why aren't you using the default CMDlets that come with Powershell (unless using PS2 or lower)?
Invoke-Webrequest and Invoke-Restmethod both support the -credential switch which can be used to authenticate against the webserver.
You can also use Get-credential to store your credentials (though you can't get yhem from a file or string that way, at least not the password)
Script (in pseudo code and since i'm on my phone no code tags) would look like this :
Invoke-Restmethod -uri 'http://bla/api' -credential (get-credential)
I am working in a script, where I am able to browse the web content or the 'url' but I am not able to copy the web content in it & download as a file.
This is what I have made so far:
$url = "http://sp-fin/sites/arindam-sites/_layouts/xlviewer.aspx?listguid={05DA1D91-F934-4419-8AEF-B297DB81A31D}&itemid=4&DefaultItemOpen=1"
$ie=new-object -com internetexplorer.application
$ie.visible=$true
$ie.navigate($url)
while($ie.busy) {start-sleep 1}
How can I copy the content of $url and save it to local drive as a file?
Update:
I got these errors:
Exception calling "DownloadFile" with "2" argument(s): "The remote server returned an error: (401) Unauthorized." At :line:6 char:47 + (New-Object system.net.webclient).DownloadFile( <<<< "$url/download-url-content", 'save.html' )
Missing ')' in method call. At :line:6 char:68 + (New-Object system.net.webclient).DownloadFile( "$url", 'save.html' <<<<
Exception calling "DownloadFile" with "2" argument(s): "The remote server returned an error: (401) Unauthorized." At :line:6 char:47 + (New-Object system.net.webclient).DownloadFile( <<<< "$url", 'save.html' )
Ok, let me explain more, on what I am trying to do: I have a excel file in our share point site & this is the file I am trying to download locally(any format), which is a part of the script, so that for the later part of the script, I can compare this file with other data & get an output.
Now if I can somehow map "my documents" from the site & able to download the file, that will also work for me.
Update Jan 2014: With Powershell v3, released with Windows 8, you can do this:
(Invoke-webrequest -URI "http://www.kernel.org").Content
Original Post, valid for Powershell Version 2
This solution is very similar to the other answers from stej, Jay Bazusi and Marco Shaw.
It is a bit more general, by installing a new module into your module directory, psurl. The module psurl adds new commands in case you have to do a lot of html-fetching (and POSTing) with powershell.
(new-object Net.WebClient).DownloadString("http://psget.net/GetPsGet.ps1") | iex
See the homepage of the code-sharing website http://psget.net/.
This nice line of PowerShell script will dowload GetPsGet.ps1 and send
it to Invoke-Expression to install PsGet Module.
Then install PsUrl, a Powershell Module inspired by curl:
To install something (in our case PsUrl) from central directory just type:
install-module PsUrl
get-module -name psurl
Output:
ModuleType Name ExportedCommands
---------- ---- ----------------
Script psurl {Get-Url, Send-WebContent, Write-Url, Get-WebContent}
Command:
get-command -module psurl
Output:
CommandType Name Definition
----------- ---- ----------
Function Get-Url ...
Function Get-WebContent ...
Alias gwc Get-WebContent
Function Send-WebContent ...
Alias swc Send-WebContent
Function Write-Url ...
You need to do this only once.
Note that this error might occur:
Q: Error "File xxx cannot be loaded because the execution of scripts is disabled on this system. Please see "get-help about_signing" for more details."
A: By default, PowerShell restricts execution of all scripts. This is all about security. To "fix" this run PowerShell as Administrator and call
Set-ExecutionPolicy RemoteSigned
From now on, in your new powershell sessions/scripts, do this:
import-module psurl
get-url "http://www.google.com"
To download and save to a file, do this:
get-url "http://www.google.com" | out-file -filepath "myfile.html"
As I understand it, you try to use IE because if automatically sends your credentials (or maybe you didn't know of any other option).
Why the above answers don't work is because you try to download file from SharePoint and you send an unauthenticated request. The response is 401.
This works:
PS>$wc=new-object system.net.webclient
PS>$wc.UseDefaultCredentials = $true
PS>$wc.downloadfile("your_url","your_file")
if the the current user of Posh has rights to download the file (is the same as the logged one in IE).
If not, try this:
PS>$wc=new-object system.net.webclient
PS>$wc.Credentials = Get-Credential
PS>$wc.downloadfile("your_url","your_file")
If you just want to download web content, use
(New-Object System.Net.WebClient).DownloadFile( 'download url content', 'save.html' )
I'm not aware of any way to save using that interface.
Does this render the page properly:
PS>$wc=new-object system.net.webclient
PS>$wc.downloadfile("your_url","your_file")
As already answered in https://stackoverflow.com/a/35202299/4636579, but with a mandatory Proxy and the credentials. Without proxy, it would be:
$url="http://aaa.bbb.ccc.ddd/rss.xml"
$WebClient = New-Object net.webclient
$path="C:\Users\hugo\xml\test.xml"
$WebClient.DownloadFile($url, $path)
$web = New-Object Net.WebClient
$web | Get-Member
$content=$web.DownloadString("http://www.bing.com")
If you're truly only concerned with the raw string content, the best route, as mentioned by a few others, is using the constructs within .NET to do this. However, I think in the previous answers a few opportunities are missed.
It's often best to use WebRequest over WebClient as it provides better control over the entire request cycle
Response buffering via System.IO.StreamReader, made possible by using WebRequest
Creating a testable, reusable tool. Which is the very nature and purpose of PowerShell
function Get-UrlContent {
<#
.SYNOPSIS
High performance url fetch
.DESCRIPTION
Given a url, will return raw content as string.
Uses:
System.Net.HttpRequest
System.IO.Stream
System.IO.StreamReader
.PARAMETER Url
Defines the url to download
.OUTPUTS
System.String
.EXAMPLE
PS C:\> Get-UrlContent "https://www.google.com"
"<!doctype html>..."
#>
[cmdletbinding()]
[OutputType([String])]
param(
[Parameter(Mandatory, ValueFromPipeline)]
[ValidateNotNullOrEmpty()]
[string] $Url)
Write-Debug "`n----- [Get-UrlContent]`n$url`n------`n`n"
$req = [System.Net.WebRequest]::CreateHttp($url)
try {
$resp = $req.GetResponse()
}
catch {
Write-Debug "`n------ [Get-UrlContent]`nDownload failed: $url`n------`n"
}
finally {
if ($resp) {
$st = $resp.GetResponseStream()
$rd = [System.IO.StreamReader]$st
$rd.ReadToEnd()
}
if ($rd) { $rd.Close() }
if ($st) { $st.Close() }
if ($resp) { $resp.Close() }
}
}