I want to write an integration test for a library that handles streamed http responses that are chunked and gzipped.
Are there any services or other suggestions I could use to do this?
Ideal endpoint or service would have:
Free
Chunked
No authentication needed
Sending at least 5k of gzipped content
The Twitter API would be perfect if it didn't require an account.
Maybe I'm over-thinking this, please say so and I'll eat my humble pie.
Looks like github.com returns chunked gzip. Example:
curl --compressed -v https://github.com/madler/zlib/commits/develop > /dev/null
...
< Server: nginx/1.0.13
< Date: Sun, 01 Apr 2012 21:32:28 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Status: 200 OK
< ETag: "bae161c1cbcaa8882f8b5c6cb1d1d270"
< X-Frame-Options: deny
< X-Runtime: 107
< Set-Cookie: _gh_sess=BAh7BzoPc2Vzc2lvbl9pZCIlZjhkOWYxYzUzMzhjMTIxZTRmYjUyZTY2MWY0YWVjZGI6EF9jc3JmX3Rva2VuIjFUK2dFR1podXlTMjcralpLUmkyTkpYZzE5ZzBLS3BLODl0OTE0VEt0ZGdJPQ%3D%3D--95e44cf6b548b93a682fdefcbca6b84cb034ca8b; path=/; expires=Sat, 01-Jan-2022 00:00:00 GMT; secure; HttpOnly
< Cache-Control: private, max-age=0, must-revalidate
< Strict-Transport-Security: max-age=2592000
< Content-Encoding: gzip
<
Related
I am using traefik in kubernetes and I have a service deployed that is returning 103 Early Hint. I can confirm that it is working by directly querying the service, e.g.
curl -D - http://contra-web-app
HTTP/1.1 103 Early Hints
Link: <https://builds.contra.com>; rel="preconnect"; crossorigin
Link: <https://fonts.googleapis.com/css2?family=Inter:wght#400;500;600;700;900&display=swap>; rel="preload"; as="font"
Link: <https://builds.contra.com/3f509d0cc/assets/entry-client-routing.4f895d55.js>; rel="modulepreload"; as="script"; crossorigin
Link: <https://www.googletagmanager.com/gtag/js?id=G-96H5NXQ2PR>; rel="preload"; as="script"
HTTP/1.1 200 OK
cache-control: no-store
referrer-policy: strict-origin-when-cross-origin
x-frame-options: sameorigin
content-type: text/html
content-length: 9062
Date: Tue, 26 Jul 2022 20:34:19 GMT
Connection: keep-alive
Keep-Alive: timeout=72
However, requesting the same service through Traefik just returns 200 response:
curl -H 'host: contra.com' -D - http://contra-traefik.traefik/gajus
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Length: 11441
Content-Type: text/html
Date: Tue, 26 Jul 2022 19:51:48 GMT
Referrer-Policy: strict-origin-when-cross-origin
Set-Cookie: contra_web_app_service=394e7e912ad85b66; Path=/; Secure
Vary: Accept-Encoding
X-Frame-Options: sameorigi
At this point, I am unable to establish whether I am missing a configuration or if Traefik does not support it.
We recently started using Azure CDN however some users reported an issue and we got a screenshot from one of them:
:
We were not able to get any more information on the issue. Any idea what could cause this and how to fix it?
Our origin server returns correctly the file with the following response headers:
Accept-Ranges: bytes
Access-Control-Allow-Origin: https://www.google.com
Cache-Control: max-age=31536000
Content-Encoding: gzip
Content-Length: 1956119
Content-Type: text/css
Date: Tue, 16 Feb 2021 18:44:10 GMT
ETag: "011889dd7ffd61:0"
Expect-CT: max-age=604800, enforce,
Feature-Policy: autoplay 'none'; camera 'none'
Last-Modified: Wed, 10 Feb 2021 18:07:38 GMT
strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
I got a similar problem. I was able to fix this by enabling cache i Dev Tools.
Enable cahce in Dev Tools
I'm watching 392 repositories on Github. However, the Github API only returns 100. Does anyone have any idea why?
https://github.com/api/v2/json/repos/watched/trivektor
You need to paginate manually using the page parameter. The HTTP Response headers will tell you the next and the last page, if available. Check the headers:
X-Next
X-Last
Examples:
curl -D- https://github.com/api/v2/json/repos/watched/trivektor
HTTP/1.1 200 OK
Server: nginx/1.0.4
Date: Sat, 22 Oct 2011 08:24:45 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Status: 200 OK
X-RateLimit-Limit: 60
ETag: "c597e396e9f17b91c5c5a7e462ba954f"
X-Next: https://github.com/api/v2/json/repos/watched/trivektor?page=2
X-Last: https://github.com/api/v2/json/repos/watched/trivektor?page=5
Now the 2nd page:
curl -D- https://github.com/api/v2/json/repos/watched/trivektor?page=2
HTTP/1.1 200 OK
Server: nginx/1.0.4
Date: Sat, 22 Oct 2011 08:28:08 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Status: 200 OK
X-RateLimit-Limit: 60
ETag: "c57d0e97e2062672cb3771467cf2abc7"
X-Next: https://github.com/api/v2/json/repos/watched/trivektor?page=3
X-Last: https://github.com/api/v2/json/repos/watched/trivektor?page=5
X-Frame-Options: deny
X-RateLimit-Remaining: 58
X-Runtime: 353ms
Content-Length: 44966
Cache-Control: private, max-age=0, must-revalidate
And the last one:
curl -D- https://github.com/api/v2/json/repos/watched/trivektor?page=5
HTTP/1.1 200 OK
Server: nginx/1.0.4
Date: Sat, 22 Oct 2011 08:28:30 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Status: 200 OK
X-RateLimit-Limit: 60
ETag: "11ce44ebc229eab0dc31731b39e10dcf"
X-Frame-Options: deny
X-RateLimit-Remaining: 57
X-Runtime: 93ms
Content-Length: 7056
Cache-Control: private, max-age=0, must-revalidate
Very common for API's to limit the size of a response object to protect against outliers. Given that it's returning a round number, that suggests this is by design. I don't see them discussing paging in their docs, so it might just be a hard cap. Either way, you should just ping github.
I am programmatically creating test accounts, and then immediately trying to log in w/ them using a selenium driven browser. Unfortunately, the browser is just redirected to the facebook homepage. I can briefly see what appears to be the correct url prior to the redirect flash by, so I have no reason to believe the browser isn't going where I intend it to.
That said, if create a fake account, and then just paste the login_url into a browser, things work fine. Anyone have any idea why that might be unique about using Selenium here? Is there anything I need to do to prepare the browser for https connections or anything?
All I'm doing is this: (using capybara and the Selenium web driver)
visit #fake_user.login_url
https://www.facebook.com/platform/test_account_login.php?user_id=100002152974488&n=ILRvb8Lqf2cq05t
GET /platform/test_account_login.php?user_id=100002152974488&n=ILRvb8Lqf2cq05t HTTP/1.1
Host: www.facebook.com
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.15) Gecko/20110303 Firefox/3.6.15
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
HTTP/1.1 302 Found
Cache-Control: private, no-cache, no-store, must-revalidate
Expires: Sat, 01 Jan 2000 00:00:00 GMT
Location: http://www.facebook.com/
P3P: CP="Facebook does not have a P3P policy. Learn why here: http://fb.me/p3p"
Pragma: no-cache
Set-Cookie: datr=d3J_TWSAN5uIXyh94O1YJkJ8; expires=Thu, 14-Mar-2013 14:06:47 GMT; path=/; domain=.facebook.com; httponly
Set-Cookie: lsd=-Lv-N; path=/; domain=.facebook.com
Content-Type: text/html; charset=utf-8
X-Powered-By: HPHP
X-FB-Server: 10.52.145.67
X-Cnection: close
Date: Tue, 15 Mar 2011 14:06:47 GMT
Content-Length: 0
http://www.facebook.com/
GET / HTTP/1.1
Host: www.facebook.com
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.15) Gecko/20110303 Firefox/3.6.15
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Cookie: datr=d3J_TWSAN5uIXyh94O1YJkJ8; lsd=-Lv-N
HTTP/1.1 200 OK
Cache-Control: private, no-cache, no-store, must-revalidate
Expires: Sat, 01 Jan 2000 00:00:00 GMT
P3P: CP="Facebook does not have a P3P policy. Learn why here: http://fb.me/p3p"
Pragma: no-cache
Set-Cookie: reg_fb_gate=http%3A%2F%2Fwww.facebook.com%2F; path=/; domain=.facebook.com
Set-Cookie: reg_fb_ref=http%3A%2F%2Fwww.facebook.com%2F; path=/; domain=.facebook.com
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
X-Powered-By: HPHP
X-FB-Server: 10.52.163.25
X-Cnection: close
Transfer-Encoding: chunked
Date: Tue, 15 Mar 2011 14:06:47 GMT
Visit Facebook home page before trying to visit login url:
visit "https://www.facebook.com"
visit #fake_user.login_url
I haven't checked the headers, but I guess Facebook sets some cookies that are needed to log in.
Umm, ok, after sending some data to the server, noting this particular part:
"Accept-Encoding: gzip,deflate\r\n"
I am getting the following response:
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 09 Apr 2010 23:25:27 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Powered-By: PHP/5.2.8
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Fri, 09 Apr 2010 23:25:27 GMT
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Pragma: no-cache
Content-Encoding: gzip
Vary: Accept-Encoding
7aa
??U-?Rh?%?2?w??PM]??7?qZ?K?)???2?&??m???"q??/p9w?????x?[`tA!G???G?5z??????a>k????????Q
???N??
('??f?,(??Y:5B???-?)?3x^0e:j?`,???**???F>G)?2????#???b??????A?k???Ar?n?
But how do I decompress it? Note that I am using the Socket Class to do all the work.
I know how to decompress it, but the problem here lies in the fact that I cannot separate the Packet from the GZipped data, psuedo-psuedocode (or whatever) on how I do it:
Socket sends packet;
Socket reads response from server, stores into a ByteArray;
Create MemoryStream, use ByteArray;
Create GZipStream, use Memorystream;
now the problem occurs; I am getting the following Error:
System.IO.InvalidDataException
The magic number in GZip header is not
correct. Make sure you are passing in
a GZip stream.
I hope the explanation is clear enough >__>.
The content that you need to gzip begins after a double-newline, and goes for (if present) Content-Length bytes. You might start there.
Is it possible for you to use System.Web.HttpRequest? It will take care of a lot of the administrative details of parsing the response for you...
Or do you have to use raw Sockets?