New-WebServiceProxy failing to authenticate with NTLM - powershell

Question

I'm dealing with a rather peculiar issue. We have a need to hit the Lists service on our SharePoint farm. Web authentication federated through an Oracle SSO, but we do have accounts configured for automation that can perform web requests. Using AAM, we have an "internal" URL configured for server side automation that bypasses directly to AD, and everything else gets pushed to the SSO.
Here's the code (sanitized) that I'm using to try to get the list collection.
$username = "DOMAIN\username"
$password = "somepassword"
$site = "https://sp.biz.com/sites/SiteCollection"
$credentials = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $username, (ConvertTo-SecureString $password -AsPlainText -Force)
$proxy = New-WebServiceProxy -Uri "$site/_vti_bin/Lists.asmx" -Credentials $credentials
$proxy.GetListCollection()
I'm hit with a 403 when I use that code.
Exception calling "GetListCollection" with "0" argument(s): "Server was unable to process request. ---> Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))"
If I change $site to use the internal URL (set via AAM) and run that on one of the front ends, I receive the list collection successfully. Now, at first I thought there was an issue with the account and permissions, but after running a Fiddler capture I see it not authenticating at all.
When I run the following cURL command, it authenticates and returns the list collection. Soap.xml is just the basic GetListCollection packet copied straight from the WDSL.
curl -v -u 'username':'pass' --ntlm -X POST -H "Content-Type: text/xml" --data-binary #soap.xml https://sp.biz.com/sites/SiteCollection/_vti_bin/Lists.asmx
Here's the sanitized verbose output from cURL.
* STATE: INIT => CONNECT handle 0x600056190; line 1029 (connection #-5000)
* Hostname was NOT found in DNS cache
* Trying <IPv6>...
* STATE: CONNECT => WAITCONNECT handle 0x600056190; line 1082 (connection #0)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to sp.biz.com (<IPv6>) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: /usr/ssl/certs/ca-bundle.crt
CApath: none
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* STATE: WAITCONNECT => PROTOCONNECT handle 0x600056190; line 1222 (connection #0)
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using TLSv1.2 / DES-CBC3-SHA
* SSL certificate verify ok.
* STATE: PROTOCONNECT => DO handle 0x600056190; line 1241 (connection #0)
* Server auth using NTLM with user 'DOMAIN\username'
> POST /sites/SiteCollection/_vti_bin/Lists.asmx HTTP/1.1
> Authorization: NTLM <snip>
> User-Agent: curl/7.39.0
> Host: sp.biz.com
> Accept: */*
> Content-Type: text/xml
> Content-Length: 0
>
* STATE: DO => DO_DONE handle 0x600056190; line 1314 (connection #0)
* STATE: DO_DONE => WAITPERFORM handle 0x600056190; line 1441 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x600056190; line 1454 (connection #0)
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 401 Unauthorized
* Server Microsoft-IIS/7.5 is not blacklisted
< Server: Microsoft-IIS/7.5
< SPRequestGuid: <snip>
< WWW-Authenticate: NTLM <snip>
< X-Powered-By: ASP.NET
< MicrosoftSharePointTeamServices: 14.0.0.7006
< X-MS-InvokeApp: 1; RequireReadOnly
< Date: Fri, 16 Jan 2015 01:02:56 GMT
< Content-Length: 0
< Set-Cookie: BIGipServerserver_pool=<snip>; expires=Sat, 17-Jan-2015 01:02:56 GMT; path=/
<
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Connection #0 to host sp.biz.com left intact
* Issue another request to this URL: 'https://sp.biz.com/sites/SiteCollection/_vti_bin/Lists.asmx'
* STATE: PERFORM => CONNECT handle 0x600056190; line 1601 (connection #-5000)
* Found bundle for host sp.biz.com: 0x60006aef0
* Re-using existing connection! (#0) with host sp.biz.com
* Connected to sp.biz.com (<IPv6>) port 443 (#0)
* STATE: CONNECT => DO handle 0x600056190; line 1075 (connection #0)
* Server auth using NTLM with user 'DOMAIN\username'
> POST /sites/SiteCollection/_vti_bin/Lists.asmx HTTP/1.1
> Authorization: NTLM <snip>
> User-Agent: curl/7.39.0
> Host: sp.biz.com
> Accept: */*
> Content-Type: text/xml
> Content-Length: 353
>
} [data not shown]
* upload completely sent off: 353 out of 353 bytes
* STATE: DO => DO_DONE handle 0x600056190; line 1314 (connection #0)
* STATE: DO_DONE => WAITPERFORM handle 0x600056190; line 1441 (connection #0)
* STATE: WAITPERFORM => PERFORM handle 0x600056190; line 1454 (connection #0)
* HTTP 1.1 or later with persistent connection, pipelining supported
< HTTP/1.1 200 OK
< Cache-Control: private, max-age=0
< Content-Type: text/xml; charset=utf-8
* Server Microsoft-IIS/7.5 is not blacklisted
< Server: Microsoft-IIS/7.5
< SPRequestGuid: <snip>
< Set-Cookie: FedAuth=<snip>; expires=Fri, 16-Jan-2015 08:36:07 GMT; path=/; secure; HttpOnly
< X-SharePointHealthScore: 0
< X-AspNet-Version: 2.0.50727
< Persistent-Auth: true
< X-Powered-By: ASP.NET
< MicrosoftSharePointTeamServices: 14.0.0.7006
< X-MS-InvokeApp: 1; RequireReadOnly
< Date: Fri, 16 Jan 2015 01:02:56 GMT
< Content-Length: 104088
< Vary: Accept-Encoding
<
{ [data not shown]
* STATE: PERFORM => DONE handle 0x600056190; line 1626 (connection #0)
100 101k 100 101k 100 353 219k 762 --:--:-- --:--:-- --:--:-- 219k
* Connection #0 to host sp.biz.com left intact
Any assistance is greatly appreciated. I'm not opposed to a C# solution over PowerShell if the cmdlets are lacking.
01-16-2015 12:13PM EST Update - I updated the question to reflect HighlyUnavailable's suggestion and included headers from the Fiddler capture.
Here are the sanitized headers from the PowerShell script:
CONNECT sp.biz.com:443 HTTP/1.1
Host: sp.biz.com
Connection: Keep-Alive
HTTP/1.1 200 Connection Established
FiddlerGateway: Direct
StartTime: 12:14:46.372
Connection: close
------------------------------------------------------------------
GET https://sp.biz.com/sites/SiteCollection/_vti_bin/Lists.asmx HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 2.0.50727.5485)
Host: sp.biz.com
Connection: Keep-Alive
HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/7.5
SPRequestGuid: <snip>
X-SharePointHealthScore: 0
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 14.0.0.7006
X-MS-InvokeApp: 1; RequireReadOnly
Date: Fri, 16 Jan 2015 17:14:46 GMT
Connection: keep-alive
Content-Length: 9066
Set-Cookie: BIGipServerserver_pool=<snip>; expires=Sat, 17-Jan-2015 17:14:46 GMT; path=/
Vary: Accept-Encoding
------------------------------------------------------------------
GET https://sp.biz.com/_vti_bin/Lists.asmx?disco HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 2.0.50727.5485)
Host: sp.biz.com
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Server: Microsoft-IIS/7.5
SPRequestGuid: <snip>
X-SharePointHealthScore: 0
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 14.0.0.7006
X-MS-InvokeApp: 1; RequireReadOnly
Date: Fri, 16 Jan 2015 17:14:46 GMT
Connection: close
Content-Length: 747
------------------------------------------------------------------
CONNECT sp.biz.com:443 HTTP/1.1
Host: sp.biz.com
Connection: Keep-Alive
HTTP/1.1 200 Connection Established
FiddlerGateway: Direct
StartTime: 12:14:47.505
Connection: close
------------------------------------------------------------------
GET https://sp.biz.com/_vti_bin/Lists.asmx?wsdl HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 2.0.50727.5485)
Host: sp.biz.com
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Server: Microsoft-IIS/7.5
SPRequestGuid: <snip>
X-SharePointHealthScore: 0
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 14.0.0.7006
X-MS-InvokeApp: 1; RequireReadOnly
Date: Fri, 16 Jan 2015 17:14:46 GMT
Connection: close
Content-Length: 72672
Set-Cookie: BIGipServerserver_pool=<snip>; expires=Sat, 17-Jan-2015 17:14:47 GMT; path=/
Vary: Accept-Encoding
------------------------------------------------------------------
CONNECT sp.biz.com:443 HTTP/1.1
Host: sp.biz.com
Connection: Keep-Alive
HTTP/1.1 200 Connection Established
FiddlerGateway: Direct
StartTime: 12:14:48.727
Connection: close
------------------------------------------------------------------
POST https://sp.biz.com/_vti_bin/Lists.asmx HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 2.0.50727.5485)
Content-Type: text/xml; charset=utf-8
SOAPAction: "http://schemas.microsoft.com/sharepoint/soap/GetListCollection"
Host: sp.biz.com
Content-Length: 321
Expect: 100-continue
HTTP/1.1 500 Internal Server Error
Cache-Control: private
Content-Type: text/xml; charset=utf-8
Server: Microsoft-IIS/7.5
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 14.0.0.7006
X-MS-InvokeApp: 1; RequireReadOnly
Date: Fri, 16 Jan 2015 17:14:48 GMT
Content-Length: 459
Set-Cookie: BIGipServerserver_pool=686493706.47873.0000; expires=Sat, 17-Jan-2015 17:14:48 GMT; path=/
------------------------------------------------------------------
Here are the headers for the cURL command.
CONNECT sp.biz.com:443 HTTP/1.1
Host: sp.biz.com:443
User-Agent: curl/7.39.0
Connection: Keep-Alive
Content-Type: text/xml
HTTP/1.1 200 Connection Established
FiddlerGateway: Direct
StartTime: 12:21:07.928
Connection: close
------------------------------------------------------------------
POST https://sp.biz.com/sites/SiteCollection/_vti_bin/Lists.asmx HTTP/1.1
Authorization: NTLM <snip>=
User-Agent: curl/7.39.0
Host: sp.biz.com
Accept: */*
Content-Type: text/xml
Content-Length: 0
HTTP/1.1 401 Unauthorized
Server: Microsoft-IIS/7.5
SPRequestGuid: <snip>
WWW-Authenticate: NTLM <snip>
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 14.0.0.7006
X-MS-InvokeApp: 1; RequireReadOnly
Date: Fri, 16 Jan 2015 17:21:07 GMT
Content-Length: 0
Set-Cookie: BIGipServerserver_pool=<snip>; expires=Sat, 17-Jan-2015 17:21:07 GMT; path=/
Proxy-Support: Session-Based-Authentication
------------------------------------------------------------------
POST https://sp.biz.com/sites/SiteCollection/_vti_bin/Lists.asmx HTTP/1.1
Authorization: NTLM <snip>
User-Agent: curl/7.39.0
Host: sp.biz.com
Accept: */*
Content-Type: text/xml
Content-Length: 417
HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Content-Type: text/xml; charset=utf-8
Server: Microsoft-IIS/7.5
SPRequestGuid: <snip>
Set-Cookie: FedAuth=<snip>; expires=Sat, 17-Jan-2015 03:20:50 GMT; path=/; secure; HttpOnly
X-SharePointHealthScore: 0
X-AspNet-Version: 2.0.50727
Persistent-Auth: true
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 14.0.0.7006
X-MS-InvokeApp: 1; RequireReadOnly
Date: Fri, 16 Jan 2015 17:21:07 GMT
Content-Length: 66628
Vary: Accept-Encoding
------------------------------------------------------------------

Answer 1

You're mixing two fundamentally different techniques here.
$proxy = New-WebServiceProxy -Uri "$site/_vti_bin/Lists.asmx" -UseDefaultCredential
$proxy.PreAuthenticate = $TRUE
$proxy.Credentials = $credentials
UseDefaultCredential will attempt to pass your currently logged in Windows domain user to the site. However, you're setting Credentials as well. Normally, you would use -Credential $credentials (see http://technet.microsoft.com/en-us/library/hh849841.aspx )
The curl command you're running is more akin to using -Credential: -u is equivalent.
Try using something like $proxy = New-WebServiceProxy -Uri "$site/_vti_bin/Lists.asmx" -Credential $credentials instead.
If that doesn't work, please edit your question to include the headers being returned from the Oracle SSO connection - it could be that it simply isn't even asking for credentials.

Answer 2

I never ended up coming up with a solution for this, but I can explain why. In our environment we use Forms Based Authentication against our Oracle Identity Foundation SSO with SAML v1.1.
When you attempt to authenticate, it redirects you to the SSO, but the client is attempting to use NTLM against the actual Web Front Ends instead of the SSO. To make this work, you need to include the X-FORMS_BASED_AUTH_ACCEPTED: f header in your request for it to actually authenticate using NTLM against the WFE (and not the SSO).
Here's the issue: You can't add headers to New-WebServiceProxy in PowerShell (up to 4.0 -- I haven't rolled out 5 yet). The only recommendation I can make for others having issues is to follow HighlyUnavailable's suggestions, or use Invoke-WebRequest and build your SOAP calls by hand.
The only issue is that Invoke-WebRequest can chew up your encoding, so here's how I've worked around it. If anybody has a suggestion for working around the encoding issue, I'm all ears.
# Set your credentials here.
$UserName = 'BartSimpson'
$Password = '3atmMySh0rtz!'
$Domain = 'SF'
$SecurePassword = ConvertTo-SecureString -String $Password -AsPlainText -Force
$Credentials = New-Object System.Management.Automation.PSCredential (($Domain + "\" + $UserName), $SecurePassword)
# SOAP request headers and body
$BaseHeaders = #{"X-FORMS_BASED_AUTH_ACCEPTED" = 'f';
"SOAPAction" = "`"http://schemas.microsoft.com/sharepoint/soap/GetListCollection`"";
"Content-Type" = "text/xml; charset=utf-8"}
$SOAP = #"
<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
<soap:Body>
<GetListCollection xmlns="http://schemas.microsoft.com/sharepoint/soap/" />
</soap:Body>
</soap:Envelope>
"#
# Gives us a random temp file to pipe output to
$TmpFile = [System.IO.Path]::GetTempFileName()
Invoke-WebRequest -Uri $URL -Headers $BaseHeaders -Credential $Credentials -Method POST -Body $SOAP -OutFile $TmpFile
# Get the outfile with UTF8 encoding
[xml]$Result = Get-Content -Raw -Path $TmpFile -Encoding UTF8
# Remove the temporary file
Remove-Item $TmpFile
Seems like a long way to go, which it is, but it works if you insist on using PowerShell.
I switched to python-suds and was able to do what I needed to.

Answer 3

In the above code, there is extra 'S' in -Credentials.
So replace the code:
$proxy = New-WebServiceProxy -Uri "$site/_vti_bin/Lists.asmx" -Credentials $credentials
with below code:
$proxy = New-WebServiceProxy -Uri "$site/_vti_bin/Lists.asmx" -Credential $credentials