Loading SharePoint Online pages with Invoke-Webrequest from Powershell - powershell

Question

I want to be able to load SharePoint Online pages with Invoke-Webrequest from Powershell.
Can someone please show me how to successfully navigate past the login screen?

Answer 1

Here is how I get list items from SPO. You need to have "Microsoft.SharePoint.Client.SharePointOnlineCredentials" dll.
Then use the below function that I have modified for SPO.
Advantage: This will even work with older PowerShell versions. If you want to target only higher level of PowerShell then also this code will work. Optionally you can use Invoke-Webrequest instead of System.Net.HttpWebRequest and System.Net.HttpWebResponse.
function Get-ListItems {
[CmdletBinding()]
PARAM (
[Parameter(Mandatory=$true)]
[String] $URL
)
#$URL = Fix-Url $URL
$xml = Request-Rest -URL $URL
return $xml
}
function Request-Rest{
[CmdletBinding()]
PARAM (
[Parameter(Mandatory=$true)]
[String] $URL,
[Parameter(Mandatory=$false)]
[Microsoft.SharePoint.Client.SharePointOnlineCredentials] $credentials,
[Parameter(Mandatory=$false)]
[String] $UserAgent = "PowerShell API Client",
[Parameter(Mandatory=$false)]
[Switch] $JSON,
[Parameter(Mandatory=$false)]
[Switch] $Raw
)
#Create a URI instance since the HttpWebRequest.Create Method will escape the URL by default.
$URI = New-Object System.Uri($URL,$true)
#Create a request object using the URI
$request = [System.Net.HttpWebRequest]::Create($URI)
#Build up a nice User Agent
$request.UserAgent = $(
"{0} (PowerShell {1}; .NET CLR {2}; {3})" -f $UserAgent, $(if($Host.Version){$Host.Version}else{"1.0"}),
[Environment]::Version,
[Environment]::OSVersion.ToString().Replace("Microsoft Windows ", "Win")
)
if ($credentials -eq $null)
{
$request.UseDefaultCredentials = $true
}
else
{
$request.Credentials = $credentials
}
if ($PSBoundParameters.ContainsKey('JSON'))
{
$request.Accept = "application/json"
}
$request.Headers.Add("X-FORMS_BASED_AUTH_ACCEPTED", "f")
#$request.Accept = "application/json;odata=verbose"
try
{
[System.Net.HttpWebResponse] $response = [System.Net.HttpWebResponse] $request.GetResponse()
}
catch
{
Throw "Exception occurred in $($MyInvocation.MyCommand): `n$($_.Exception.Message)"
}
$reader = [IO.StreamReader] $response.GetResponseStream()
if (($PSBoundParameters.ContainsKey('JSON')) -or ($PSBoundParameters.ContainsKey('Raw')))
{
$output = $reader.ReadToEnd()
}
else
{
[xml]$output = $reader.ReadToEnd()
}
$reader.Close()
Write-Output $output
$response.Close()
}

Answer 2

Though I do not know the direct way to pass the credentials along with Invoke-WebRequest itself, one workaround I found is, capture the cookie values by manual authenticating the SharePoint page and use those for subsequent requests. You can use fiddler or some other similar tool to grab the cookies. The two cookie names were 'FedAuth' and 'rtFa'
$session = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$cookieCollection = New-Object System.Net.CookieCollection
$cookie1 = New-Object System.Net.Cookie
$cookie1.Domain = "<your domain>.sharepoint.com"
$cookie1.Name = "FedAuth"
$cookie1.Value = "<cookie value here>"
$cookieCollection.Add($cookie1)
$cookie2 = New-Object System.Net.Cookie
$cookie2.Domain = "<your domain>.sharepoint.com"
$cookie2.Name = "rtFa"
$cookie2.Value = "<cookie value here>"
$cookieCollection.Add($cookie2)
$session.Cookies.Add($cookieCollection)
$uri = "https:<your site collection here>/_layouts/15/SharePointDesignerSettings.aspx"
$response = Invoke-WebRequest -Uri $uri -WebSession $session -Method Default
$form = $response.Forms[0]
You can use $form to examine the Html elements. If you want to submit the changes made to form use the below line
$response = Invoke-WebRequest -Uri $uri -WebSession $session -Method POST -Body $form
Note: There is an issue with Invoke-WebRequest with respect to field submission.. basically it uses 'id' of input element instead of 'name' in form fields collection.. the below url has the code to convert field Id to Name
https://d-fens.ch/2013/05/11/invoke-webrequest-uses-id-attribute-of-input-element-as-field-name-in-form-fields-collection/

Answer 3

If you are looking for the final content of a page, Invoke-WebRequest will not do what you need. Much of the content of a SharePoint page is loaded asynchronously using JavaScript. Invoke-WebRequest will only return the initial HTML content from the page.
What kind of content are you looking for from the page? Most everything about SharePoint can be accessed using RESTful queries (Invoke-RESTMethod and the SharePoint REST API) or from the PowerShell SharePoint PNP and SharePoint Online cmdlet libraries.