Invoke-WebRequest returning unexpected results for some addresses - powershell

I seem to be having issues with certain addresses when trying to process with Invoke-WebRequest
The address below is one example of a few that I have come across.
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
(Invoke-WebRequest https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt).Content
(Invoke-WebRequest https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt).RawContent
The Content property seems to only return an Array of numbers. RawContent displays a result closer to what I would expect.
What is the reason for this?
Content
35
32
65
100
32
102
105
108
116
101
RawContent
HTTP/1.1 200 OK
x-amz-id-2: juI0bRXZacf19D5/sjmsw53XudBjlbSYtO0XKVQP0N+dWXvixKoWs8mV8oF23SLt8iGjaE/CKco=
x-amz-request-id: 5B68B280AD366CDE
Accept-Ranges: bytes
Content-Length: 43642
Content-Type:
Date: Fri, 13 Jul 2018 23:25:09 GMT
ETag: "0a43bd5ebd9b3e2ee08a205d163cb0ac"
Last-Modified: Wed, 09 Mar 2016 19:46:05 GMT
Server: AmazonS3
# Ad filter list by Disconnect
# License: GPLv3
# Contact: support [at] disconnect.me
htmlcontent

If you look at the headers, the Content Type is empty, therefore Invoke-WebRequest doesn't know what it's dealing with and treats the data as binary data (byte array).
(Invoke-WebRequest https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt).Headers
Key Value
--- -----
Accept-Ranges bytes
Content-Length 43642
Content-Type
Date Sat, 14 Jul 2018 02:28:36 GMT
ETag "0a43bd5ebd9b3e2ee08a205d163cb0ac"
Last-Modified Wed, 09 Mar 2016 19:46:05 GMT
Server AmazonS3
Whilst for example calling Invoke-WebRequest on this question:
(Invoke-WebRequest https://stackoverflow.com/questions/51333965/invoke-webrequest-returning-unexpecte
d-results-for-some-addresses).Headers
Key Value
--- -----
X-Frame-Options SAMEORIGIN
X-Request-Guid 9366c316-b9b3-467b-8911-b18bb4752e1c
Strict-Transport-Security max-age=15552000
Content-Security-Policy upgrade-insecure-requests
Age 0,0
Accept-Ranges bytes,bytes
Content-Length 108783
Cache-Control private
Content-Type text/html; charset=utf-8
…
EDIT: If you want to overrule the content-type you could do something like this:
$url = 'https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt'
[System.Net.HttpWebRequest]$req = [System.Net.WebRequest]::Create($url) -as [System.Net.HttpWebRequest]
[System.Net.HttpWebResponse]$resp = $req.getResponse()
$respStream = $resp.getResponseStream()
$sr = new-object IO.StreamReader($respStream)
$result = $sr.ReadToEnd()
$resp.Close()

Related

Do comparison without triggering command with the same name

So I'm trying to do a comparison and it just happens to be that the value I'm checking for has the same name as a command on the computer with is a problem.
This is the command i run.
if $current_branch != "HEAD"; then echo '1'; fi;
And here is the result as you can see the HEAD program ran.
400 URL must be absolute
Content-Type: text/plain
Client-Date: Thu, 01 Mar 2018 16:45:48 GMT
Client-Warning: Internal response
200 OK
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Connection: keep-alive
Date: Thu, 1 Mar 2018 16:45:49 GMT
Server: PWS/8.3.1.0.8
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Access-Control-Allow-Origin: https://www.head.com
Client-Date: Thu, 01 Mar 2018 16:45:49 GMT
Client-Peer: 151.249.91.221:443
Client-Response-Num: 1
Client-SSL-Cert-Issuer: /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
Client-SSL-Cert-Subject: /C=US/ST=California/L=Campbell/O=CDNetworks Inc./CN=support2.cdnetworks.net
Client-SSL-Cipher: ECDHE-RSA-AES256-SHA
Client-SSL-Socket-Class: IO::Socket::SSL
Content-Security-Policy: frame-ancestors 'self' https://head.testing-varnish.symmetrics.de
Set-Cookie: geoip=IC; expires=Sat, 31-Mar-2018 16:45:49 GMT; Max-Age=2592000; path=/; domain=head.com
Set-Cookie: lang=en; expires=Sat, 31-Mar-2018 16:45:49 GMT; Max-Age=2592000; path=/; domain=head.com
Set-Cookie: currentLangId=1; expires=Sat, 31-Mar-2018 16:45:49 GMT; Max-Age=2592000; path=/; domain=head.com
Set-Cookie: root=1; expires=Sat, 31-Mar-2018 16:45:49 GMT; Max-Age=2592000; path=/; domain=head.com
X-Frame-Options: SAMEORIGIN
X-Px: nc h0-s42.p1-arn ( h0-s4012.p6-lhr>CONN), nc h0-s4012.p6-lhr ( h0-s4102.p11-fra), nc h0-s4102.p11-fra ( origin)
So how do i avoid this from happening and not have command collide with the string value i want to compare?
The argument to if is always a command. You are probably looking for the special command [ (aka test) which compares strings.
if [ "$current_branch" != "HEAD" ]; then
echo '1'
fi
Take note of the requirement to have spaces on both sides of the [ and ] tokens, and the requirement to quote strings which might contain shell metacharacters.
Depending on whether you require pre-POSIX compatibility, you might also want to invert the negation. On the other hand, in modern shells like Bash and Ksh, you probably want to use the replacement [[ which is somewhat more robust and versatile.
The requirement to have a terminating ] for symmetry is purely aesthetic, but still a requirement. It's pretty quirky that the original Bourne shell didn't have string comparison features built into the shell itself, but, well, this is what we ended up with.
As an aside, if your example is representative of your actual code, you might want to use the somewhat more parsimonious shorthand syntax
[ "$current_branch" = "HEAD" ] || echo '1'

What could cause a 302 redirect?

I have a wordpress site and I have a blog. When I try to share my blog post on FB http://videoefekt.cz/?p=472 , it shows the thumb of my homepage. I tried to change permalinks to clear, deactivate plugins and chceck .htaccess file, everything looks fine, but the FB sharing debugger shows I have 302 HTTP Redirect to my homepage, can anyone tell me what could cause this problem ?
Simulate the facebook crawler access to your site and it made redirect to home page when using ipv6 address. Possibly you have misconfig when the request is from ipv6 address.
curl -H "User-Agent: facebookexternalhit/1.1" "http://videoefekt.cz/?p=472" -vvvv -6
* About to connect() to videoefekt.cz port 80 (#0)
* Trying 2a02:2b88:1:4::a6...
* Connected to videoefekt.cz (2a02:2b88:1:4::a6) port 80 (#0)
> GET /?p=472 HTTP/1.1
> Host: videoefekt.cz
> Accept: */*
> User-Agent: facebookexternalhit/1.1
>
< HTTP/1.1 302 Found
< Date: Thu, 04 Jan 2018 04:16:32 GMT
< Server: Apache
< Pragma: no-cache
< Expires: Wed, 11 Jan 1984 05:00:00 GMT
< Cache-Control: no-cache, must-revalidate, max-age=0
< Link: <http://videoefekt.cz/wp-json/>; rel="https://api.w.org/"
< Set-Cookie: _icl_current_language=cs; expires=Fri, 05-Jan-2018 04:16:33 GMT; Max-Age=86400; path=/
< Set-Cookie: wfvt_3708104896=5a4daaa1e2090; expires=Thu, 04-Jan-2018 04:46:33 GMT; Max-Age=1800; path=/; HttpOnly
< Set-Cookie: PHPSESSID=8a1a1d9247ac9d9707b616c6c0ad2ccc; path=/
< Upgrade: h2,h2c
< Connection: Upgrade
< Location: http://videoefekt.cz/
< Vary: User-Agent
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
<
* Connection #0 to host videoefekt.cz left intact

Empty response from Bitstamp.net

I am trying to use the private api from Bitstamp to query for my account's balance. I believe I am posting the right data as no error is returned but Bitstamp's answer is always empty
* Trying 107.154.249.133...
* Connected to www.bitstamp.net (107.154.249.133) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection:
ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* NPN, negotiated HTTP1.1
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: OU=Domain Control Validated; CN=*.bitstamp.net
* start date: Apr 8 14:11:39 2015 GMT
* expire date: Apr 8 15:48:13 2018 GMT
* subjectAltName: www.bitstamp.net matched
* issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.;
OU=http://certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate
Authority - G2
* SSL certificate verify ok.
> POST /api/v2/balance HTTP/1.1
Host: www.bitstamp.net
User-Agent: Bitstamp C++ API Client
Accept: */*
Content-Length: 134
Content-Type: application/x-www-form-urlencoded
* upload completely sent off: 134 out of 134 bytes
< HTTP/1.1 301 MOVED PERMANENTLY
< Access-Control-Allow-Headers: x-requested-with, Content-Type, origin, accept, cache-control
< Access-Control-Allow-Methods: POST, GET
< Access-Control-Allow-Origin: *
< Content-Language: en
< Content-Type: text/html; charset=utf-8
< Date: Fri, 08 Sep 2017 17:46:41 GMT
< Location: http://www.bitstamp.net/api/v2/balance/
< Server: Apache
< Strict-Transport-Security: max-age=63072000; includeSubDomains
< Vary: Accept-Language
< X-Frame-Options: SAMEORIGIN
< Content-Length: 0
< Connection: keep-alive
< Set-Cookie: visid_incap_99025=wHhR5uPzT92ReB9XbeNToIHXslkAAAAAQUIPAAAAAAAbNQJEog5ZojcqdOv0dFhd; expires=Sat, 08 Sep 2018 10:32:13 GMT; path=/; Domain=.bitstamp.net
< Set-Cookie: nlbi_99025=VyBXXwUOsQHckThX8F1n9AAAAABOW3tLZ0nUHyp47yinIimv; path=/; Domain=.bitstamp.net
< Set-Cookie: incap_ses_569_99025=04PeXe3KGFAAVw//8n7lB4HXslkAAAAA09BkGnsaiskl2dkFxzhFXQ==; path=/; Domain=.bitstamp.net
< X-Iinfo: 9-15073063-15060114 PNNN RT(1504892800858 31) q(0 0 0 -1) r(2 2) U6
< X-CDN: Incapsula
<
* Connection #0 to host www.bitstamp.net left intact
result: 0
response:
terminate called after throwing an instance of 'std::runtime_error'
what(): Bitstamp response doesn't contain result data
Any idea ?
Thanks a lot in advance !
Initially I had the balance path to be /balance, but after changing to /balance/ it worked and got 200 back with payload.
So instead of http://www.bitstamp.net/api/v2/balance use http://www.bitstamp.net/api/v2/balance/, note the trailing backslash

Coldfusion: CFHTTP landing page redirection

I am working on a project that requires a CFHTTP call to book publishers that returns an excerpt from their site. My issue is that one of the publisher's URLS is converted to another and I can't seem to figure out how to get the returned information from them.
For example: The url I would visit is http://www.pubsite.com/bookID/99999
When I go to that site in the browser, it automatically redirects me to: http://www.pubsite.com/publisherID/12345
This is fine and I can see the information I need. However, when I try to automate this process using Coldfusion, I am stuck. There is no way to know what that publisherID is ahead of time.
My code is this:
<cfset thisURL = "http://www.pubsite.com/bookID/#Trim(arguments.BID)#">
<cfhttp method="get" url="#thisURL#" useragent="#CGI.http_user_agent#" result="objGet" redirect="true" resolveurl="true"/>
<cfdump var="#objGet#" />
Nothing in the returned dump (header or filecontent) indicates the publisherID, or the true landing URL, and I get a ResponseCode of 200 so I don't know what to do next.
After much Google searching, I'm stumped. Thanks for any insight!
EDIT TO INCLUDE DUMP:
struct
Charset UTF-8
ErrorDetail [empty string]
Filecontent
binary
6033687967848980
[truncated]
Header HTTP/1.1 200 OK X-Firenze-Processing-Time: 422.785 X-Firenze-Processing-Time: 626.552 X-Varnish-Cache: miss X-SmartBan-URL: /content/1177/0959683607085571 X-Firenze-Processing-Times: detect-robot: 0.315 X-Firenze-Processing-Times: add-external-dependencies-info: 0.033 X-Firenze-Processing-Times: content-pre-retrieve: 307.973 X-Firenze-Processing-Times: add-req-info: 10.427 X-Firenze-Processing-Times: authn-authz: 48.986 X-Firenze-Processing-Times: ac-info-request: 4.702 X-Firenze-Processing-Times: login-check: 2.104 X-Firenze-Processing-Times: last-authorized-identity: 1.769 X-Firenze-Processing-Times: retrieve-current-issue: 0.038 X-Firenze-Processing-Times: retrieve-content-page-content: 8.148 X-Firenze-Processing-Times: retrieve-content-page-prev: 3.260 X-Firenze-Processing-Times: retrieve-content-page-next: 3.310 X-Firenze-Processing-Times: retrieve-ads: 9.608 X-Firenze-Processing-Times: retrieve-session-data: 4.577 X-Firenze-Processing-Times: log-builder: 3.302 X-Firenze-Processing-Times: servlet: 207.151 Accept-Ranges: bytes X-XXXX-RequestId: VPTH5atDd2QAABNxM48AAAD- Vary: Accept-Encoding Connection: close X-XXXX-SessionId: m3H5rW2RVOiJ7qb3WTMt0w Set-Cookie: JSESSIONID=FF1AA7D6946D968D5E100C8FD1B8205C.wa126.XXXX.org; Path=/ Age: 0 Content-Type: text/html;charset=UTF-8 Server: nginx/0.7.67 X-Varnish: 2761793264 X-Varnish-Hostname: varnish8.XXXX.org X-SmartBan-Host: hol.XXXX.com x-XXXX-user: memno=1000480925; name=XXXX; type=INST; ip_access=true; password_access=true Date: Mon, 02 Mar 2015 20:28:21 GMT Via: 1.1 varnish
Mimetype text/html
Responseheader
struct
Accept-Ranges bytes
Age 0
Connection close
Content-Type text/html;charset=UTF-8
Date Mon, 02 Mar 2015 20:28:21 GMT
Explanation OK
Http_Version HTTP/1.1
Server nginx/0.7.67
Set-Cookie JSESSIONID=FF1AA7D6946D968D5E100C8FD1B8205C.wa126.XXXX.org; Path=/
Status_Code 200
Vary Accept-Encoding
Via 1.1 varnish
X-Firenze-Processing-Time
struct
1 422.785
2 626.552
X-Firenze-Processing-Times
struct
1 detect-robot: 0.315
10 retrieve-content-page-content: 8.148
11 retrieve-content-page-prev: 3.260
12 retrieve-content-page-next: 3.310
13 retrieve-ads: 9.608
14 retrieve-session-data: 4.577
15 log-builder: 3.302
16 servlet: 207.151
2 add-external-dependencies-info: 0.033
3 content-pre-retrieve: 307.973
4 add-req-info: 10.427
5 authn-authz: 48.986
6 ac-info-request: 4.702
7 login-check: 2.104
8 last-authorized-identity: 1.769
9 retrieve-current-issue: 0.038
X-XXXX-RequestId VPTH5atDd2QAABNxM48AAAD-
X-XXXX-SessionId m3H5rW2RVOiJ7qb3WTMt0w
X-SmartBan-Host hol.XXXX.com
X-SmartBan-URL /content/1177/0959683607085571
X-Varnish 2761793264
X-Varnish-Cache miss
X-Varnish-Hostname varnish8.XXXX.org
x-XXXX-user memno=1000480925; name=XXXX; type=INST; ip_access=true; password_access=true
Statuscode 200 OK
Text YES

Filter invoices by Customer ID PHP Devkit

I'm trying to filter invoices that I'm getting using QuickBook IPP and PHP devkit, but for some reason, quickbooks keeps sending back an empty result. My code is below:
$query = '<ContactIdSet><Id>340</Id></ContactIdSet>';
$invoices = $invoiceService->findAll($quickbooks->Context, $quickbooks->creds['qb_realm'], $query, 1, 999);
With this code, my request (minus Oauth info) is:
POST https://services.intuit.com/sb/invoice/v2/538361885 HTTP/1.1
Content-Type: text/xml
Authorization: OAuth realm="", oauth_signature_method="HMAC-SHA1", oauth_signature="", oauth_nonce="GAmtH", oauth_timestamp="1366692972", oauth_token="", oauth_consumer_key="", oauth_version="1.0"
Content-Length: 298
<?xml version="1.0" encoding="UTF-8"?>
<InvoiceQuery xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.intuit.com/sb/cdm/v2">
<StartPage>1</StartPage>
<ChunkSize>999</ChunkSize>
<ContactIdSet>
<Id>340</Id>
</ContactIdSet>
</InvoiceQuery>
but even though there are invoices under that customer id, the response has no invoices:
HTTP/1.1 200 OK
Date: Tue, 23 Apr 2013 05:06:39 GMT
Server: Apache/2.2.22 (Unix)
ContextInitTime: 9
Response-Id: cb8b6eaa-f810-4fad-9f69-3485f400680d
IdRestateTime: 4
HeaderQueryTime: 4
Content-Encoding: gzip
HeaderDAOTime: 4
Response-App-Time: 62
Response-Routing-Time: 67
Content-Length: 133
Connection: close
Content-Type: text/xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><RestResponse xmlns="http://www.intuit.com/sb/cdm/v2"><Invoices/></RestResponse>
Any ideas as to why quickbooks won't return the invoices that are associated with that customer?
Here is the XML response that shows the invoice exists when I use ->findAll()
HTTP/1.1 200 OK
Date: Tue, 23 Apr 2013 17:44:02 GMT
Server: Apache/2.2.22 (Unix)
LineQueryTime: 35
ContextInitTime: 94
Response-Id: 9b3d4e46-a6b5-4f90-81f4-3c42b0462330
AssemblingTime: 0
HeaderQueryTime: 91
Content-Encoding: gzip
HeaderDAOTime: 91
LineDAOTime: 35
Response-App-Time: 271
Response-Routing-Time: 276
Content-Length: 930
Connection: close
Content-Type: text/xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><RestResponse xmlns="http://www.intuit.com/sb/cdm/v2"><Invoices><Invoice><Id idDomain="NG">681891</Id><SyncToken>3</SyncToken><MetaData><CreateTime>2013-04-18T03:13:26.796Z</CreateTime><LastUpdatedTime>2013-04-23T03:49:05.800Z</LastUpdatedTime></MetaData><ExternalKey idDomain="QB">1189</ExternalKey><Synchronized>true</Synchronized><Header><DocNumber>WEB000126</DocNumber><TxnDate>2013-02-28T00:00:00Z</TxnDate><Status>Paid</Status><CustomerId idDomain="QB">340</CustomerId><CustomerName>Test, Ray & Teryl</CustomerName><RemitToId idDomain="QB">340</RemitToId><RemitToName>Test, Ray & Teryl</RemitToName><ShipDate>2013-02-28T00:00:00Z</ShipDate><SubTotalAmt>375</SubTotalAmt><TaxRate>0</TaxRate><TaxAmt>0</TaxAmt><TotalAmt>375</TotalAmt><ToBePrinted>true</ToBePrinted><ToBeEmailed>false</ToBeEmailed><ARAccountId idDomain="QB">64</ARAccountId><ARAccountName>Accounts Receivable</ARAccountName><SalesTermId idDomain="QB">8</SalesTermId><SalesTermName>Due on 1st</SalesTermName><DueDate>2013-03-01T00:00:00Z</DueDate><BillAddr><Id idDomain="QB">00000000000001cg</Id><Line1>Ray & Teryl Test</Line1><Line2>3290 Test Test</Line2><City>Stone Mountain</City><CountrySubDivisionCode>GA</CountrySubDivisionCode><PostalCode>30087</PostalCode><Default>true</Default><Tag>Billing</Tag></BillAddr><ShipAddr><Id idDomain="QB">00000000000003rW</Id><Default>false</Default><Tag>Shipping</Tag></ShipAddr><BillEmail>Test#Test.com</BillEmail><Balance>0</Balance></Header><Line><Id idDomain="QB">1191</Id><Desc>Conservatory Registration Fee</Desc><Amount>125</Amount><Taxable>false</Taxable><ItemId idDomain="QB">14</ItemId><ItemName>Artios Conservatory</ItemName><ItemType>Service</ItemType><UnitPrice>125</UnitPrice><Qty>1</Qty><SalesTaxCodeId idDomain="QB">2</SalesTaxCodeId><SalesTaxCodeName>Non</SalesTaxCodeName></Line><Line><Id idDomain="QB">1192</Id><Desc>Preparatory Registration Fee</Desc><Amount>125</Amount><Taxable>false</Taxable><ItemId idDomain="QB">142</ItemId><ItemName>Elem Prep Reg (5/1-8/31)</ItemName><ItemType>Service</ItemType><UnitPrice>125</UnitPrice><Qty>1</Qty><SalesTaxCodeId idDomain="QB">2</SalesTaxCodeId><SalesTaxCodeName>Non</SalesTaxCodeName></Line><Line><Id idDomain="QB">1193</Id><Desc>Preparatory Registration Fee</Desc><Amount>125</Amount><Taxable>false</Taxable><ItemId idDomain="QB">142</ItemId><ItemName>Elem Prep Reg (5/1-8/31)</ItemName><ItemType>Service</ItemType><UnitPrice>125</UnitPrice><Qty>1</Qty><SalesTaxCodeId idDomain="QB">2</SalesTaxCodeId><SalesTaxCodeName>Non</SalesTaxCodeName></Line><Line><Id idDomain="QB">1194</Id><Desc>PayPal Convenience Fee</Desc></Line></Invoice></Invoices></RestResponse>
To add an answer for the question, there is no support for that specific filter in v2 of the API.
You'll need to filter within your application
thanks
Jarred