Download all pdfs on a webpage with PowerShell - powershell

Question

I found online the following code to download all the pdfs on a webpage:
$psPage = Invoke-WebRequest "https://www.pi.infn.it/~rizzo/ingegneria/appunti_fisII_ing_mecc/
"
$urls = $psPage.ParsedHtml.getElementsByTagName("A") | ? {$_.href -like "*.pdf"} | Select-Object -ExpandProperty href
$urls | ForEach-Object {Invoke-WebRequest -Uri $_ -OutFile ($_ | Split-Path -Leaf)}
but PS gave me the error:
Invoke-WebRequest : The response content cannot be parsed because the Internet Explorer engine is not
available, or Internet Explorer's first-launch configuration is not complete. Specify the UseBasicParsing
parameter and try again.
At C:\Users\Raffaele\Desktop\Nuova cartella\a.ps1:1 char:11
+ $psPage = Invoke-WebRequest "https://www.pi.infn.it/~rizzo/ingegneria ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotImplemented: (:) [Invoke-WebRequest], NotSupportedException
+ FullyQualifiedErrorId : WebCmdletIEDomNotSupportedException,Microsoft.PowerShell.Commands.InvokeWebReq
uestCommand
You cannot call a method on a null-valued expression.
At C:\Users\Raffaele\Desktop\Nuova cartella\a.ps1:2 char:1
+ $urls = $psPage.ParsedHtml.getElementsByTagName("A") | ? {$_.href -li ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull
Split-Path : Cannot bind argument to parameter 'Path' because it is null.
At C:\Users\Raffaele\Desktop\Nuova cartella\a.ps1:4 char:66
+ ... h-Object {Invoke-WebRequest -Uri $_ -OutFile ($_ | Split-Path -Leaf)}
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidData: (:) [Split-Path], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.S
plitPathCommand
Invoke-WebRequest : Cannot validate argument on parameter 'Uri'. The argument is null or empty. Provide an
argument that is not null or empty, and then try the command again.
At C:\Users\Raffaele\Desktop\Nuova cartella\a.ps1:4 char:48
+ $urls | ForEach-Object {Invoke-WebRequest -Uri $_ -OutFile ($_ | Spli ...
+ ~~
+ CategoryInfo : InvalidData: (:) [Invoke-WebRequest], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.PowerShell.Commands.InvokeWebReques
tCommand
How can I overcome this problem? Why it requires internet explorer engine?
EDIT: I tried to modify the code in this way:
$site = "https://www.pi.infn.it/~rizzo/ingegneria/appunti_fisII_ing_mecc/"
$psPage = Invoke-WebRequest -Uri $site -UseBasicParsing
$urls = $psPage.ParsedHtml.getElementsByTagName("A")
$urls | where {$_.pathname -like "*pdf"} | % {Invoke-WebRequest -Uri "$site$($_.pathname)" -OutFile $_.pathname }
and the error is:
You cannot call a method on a null-valued expression.
At C:\Users\Raffaele\Desktop\Nuova cartella\a.ps1:3 char:1
+ $urls = $psPage.ParsedHtml.getElementsByTagName("A")
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull
EDIT 2 I tried to modify the code in this way. New code:
$psPage = Invoke-WebRequest -Uri -UseBasicParsing "https://www.pi.infn.it/~rizzo/ingegneria/appunti_fisII_ing_mecc/"
$urls = $psPage.ParsedHtml.getElementsByTagName("A") | ? {$_.href -like "*.pdf"} | Select-Object -ExpandProperty href
$urls | ForEach-Object {Invoke-WebRequest -Uri $_ -OutFile ($_ | Split-Path -Leaf)}
The Windows PowerShell gave me a new error:
Invoke-WebRequest : Missing an argument for parameter 'Uri'. Specify a parameter of type 'System.Uri' and
try again.
At C:\Users\Raffaele\Desktop\Nuova cartella\b.ps1:1 char:29
+ $psPage = Invoke-WebRequest -Uri -UseBasicParsing "https://www.pi.inf ...
+ ~~~~
+ CategoryInfo : InvalidArgument: (:) [Invoke-WebRequest], ParameterBindingException
+ FullyQualifiedErrorId : MissingArgument,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
You cannot call a method on a null-valued expression.
At C:\Users\Raffaele\Desktop\Nuova cartella\b.ps1:2 char:1
+ $urls = $psPage.ParsedHtml.getElementsByTagName("A") | ? {$_.href -li ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull
Split-Path : Cannot bind argument to parameter 'Path' because it is null.
At C:\Users\Raffaele\Desktop\Nuova cartella\b.ps1:3 char:66
+ ... h-Object {Invoke-WebRequest -Uri $_ -OutFile ($_ | Split-Path -Leaf)}
+ ~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidData: (:) [Split-Path], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.S
plitPathCommand
Invoke-WebRequest : Cannot validate argument on parameter 'Uri'. The argument is null or empty. Provide an
argument that is not null or empty, and then try the command again.
At C:\Users\Raffaele\Desktop\Nuova cartella\b.ps1:3 char:48
+ $urls | ForEach-Object {Invoke-WebRequest -Uri $_ -OutFile ($_ | Spli ...
+ ~~
+ CategoryInfo : InvalidData: (:) [Invoke-WebRequest], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.PowerShell.Commands.InvokeWebReques
tCommand

Answer 1

In your way : First you have to remove "about:" in your URL or replace it by nothing :
$site = "https://www.pi.infn.it/~rizzo/ingegneria/appunti_fisII_ing_mecc/"
$psPage = Invoke-WebRequest $site
$urls = $psPage.ParsedHtml.getElementsByTagName("A") | ? {$_.href -like "*.pdf"} | Select-Object -ExpandProperty href | ForEach-Object {$_.replace("about:", "")}
Second you have to recreate full URL :
$urls | ForEach-Object {Invoke-WebRequest -Uri "$site$_" -OutFile $_ }
But you can simplify using "textcontent" or "pathname" :
$site = "https://www.pi.infn.it/~rizzo/ingegneria/appunti_fisII_ing_mecc/"
$psPage = Invoke-WebRequest $site
$urls = $psPage.ParsedHtml.getElementsByTagName("A")
$urls | where {$_.pathname -like "*pdf"} | % {Invoke-WebRequest -Uri "$site$($_.pathname)" -OutFile $_.pathname }