I've a small site designed to sell a HTTP-downloadable, ~300 MB PDF, No-DRM, page-scanned images e-book (download the test copy here http://test.magicmedicine.eu/get/ac123457965d0d4b4d17557a73cf2fe8 ).
It works flawlessly on PC, Mac and Android, but I'm experiencing issues with iOS: when the customer opens (I tried via broadband Wi-Fi+DSL) the download URL via Safari, the page loads for ~45 seconds (the page is blank but the activity indicator rotates), then Safari exits with no error messages at all.
I tried to create the PDF with the "Fast web view" (=progressive download) attribute and I also lowered the compatibility to the minimum (PDF version 1.3), with no results.
Application-side, the download is sent from Apache+PHP via mod_xsendfile ( https://tn123.org/mod_xsendfile/ ) to the client with the following headers (my intent is to avoid the PDF-in-the-browser-via-plugin view):
HTTP/1.1 200 OK
Date: Wed, 23 May 2012 09:50:13 GMT
Server: Apache/2.2.15 (CentOS)
X-Powered-By: PHP/5.3.13
Expires: Thu, 24 May 2012 11:50:13 +0200
Cache-Control: must-revalidate, post-check=0, pre-check=0
Pragma: public
Content-Disposition: attachment; filename="book.pdf"
Last-Modified: Sun, 20 May 2012 11:26:54 GMT
ETag: "2e01b4-dde8a9b-4c07610070008"
Content-Length: 232688283
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: application/octet-stream
Any ideas?
Note: I asked this on SuperUser a couple of days ago and was closed as "off topic". I hope here it's ok to repost it here.
Related
My initial response headers - notice the Accept-Ranges header
HTTP/1.1 200 OK
X-Powered-By: Express
Vary: Origin
Access-Control-Allow-Credentials: true
X-RateLimit-Limit: 1
X-RateLimit-Remaining: 0
Date: Thu, 08 Apr 2021 06:14:19 GMT
X-RateLimit-Reset: 1617862461
Accept-Ranges: bytes
Content-Length: 100000000
Content-Type: text/plain; charset=utf-8
Content-Disposition: attachment; filename="some_file.txt"
Connection: keep-alive
Keep-Alive: timeout=5
I then restart the server and click resume download in chrome, but chrome doesn't send back in Range request headers
I'm following the documentation on Mozilla's website
Am I missing a header or misunderstanding how this works, especially with chrome and other browsers? Is there another way I can manually support resuming downloads by sending the right response and understanding the right request? From a technical perspective, if chrome sends back which range it now needs I will be able to resume the download.
According to this article, chrome should support something like this. I just need to be pointed in the right direction.
Ty!
Chrome needs some way to know that the file it's trying to download at that URL is indeed the same file when it tries to resume.
If you add support for an ETag header, this will likely work.
Edit: it seems the issue was caused by a dropped cookie. There should have been a session id cookie as well.
For posterity, here's the original question
When sending a request formed as this
GET https://<url>?<parameters>
Cache-Control: max-age=0
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Charset: iso-8859-1,utf-8,UTF-8
Accept-Encoding: gzip, x-gzip, deflate, x-bzip2
Accept-Language: en-US,en;q=0.5
If-None-Match: "6eb7d55abfd0546399e3245ad3a76090"
User-Agent: Mozilla/5.0 libwww-perl/6.13
Cookie: auth_token=<blah>; __cfduid=<blah>
Cookie2: $Version="1"
I receive the following response
response-type: text/html
charset=utf-8
HTTP/1.1 406 Not Acceptable
Cache-Control: no-cache
Connection: keep-alive
Date: Fri, 12 Feb 2016 18:34:00 GMT
Server: cloudflare-nginx
Content-Type: text/html; charset=utf-8
CF-RAY: 273a62969a9b288e-SJC
Client-Date: Fri, 12 Feb 2016 18:34:00 GMT
Client-Peer: <IP4>:443
Client-Response-Num: 10
Client-SSL-Cert-Issuer: /C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limite
d/CN=COMODO ECC Domain Validation Secure Server CA 2
Client-SSL-Cert-Subject: /OU=Domain Control Validated/OU=PositiveSSL Multi-Domai
n/CN=ssl<blah>.cloudflaressl.com
Client-SSL-Cipher: <some value>
Client-SSL-Socket-Class: IO::Socket::SSL
Client-SSL-Warning: Peer certificate not verified
Client-Transfer-Encoding: chunked
Status: 406 Not Acceptable
X-Runtime: 9
I'm not entirely sure why the response is 406 Not Acceptable. When
downloaded with firefox, the file in question in 996 KB (as reported
by Windows 8.1's explorer). It looks like I have a partially
transferred file from my perl script at 991 KB (again, windows
explorer size), so it got MOST of the file before throwing the Not
Acceptable response. Using the same URL pattern and request style, I
was able to successfully download a 36 MB file from the server with
this perl library and request form, so the size of the file should not
be magically past some max (chunk) size. As these files are being
updated on approximately 15-minute intervals, I suppose it's possible
that a write was performed on the server, invalidating the ETag before
all chunks were complete on this file?
I tried adding chunked to Accept-Encoding, but that's not for
Transfer encoding and it appears to have no affect on the server's behavior. Additionally, as I've been able to download larger files
(same format) from the same server, that alone shouldn't be the cause
of my woes. LWP is supposed to be able to handle chunked data
returned by a response to GET (as per this newsgroup post).
The server in question is running nginx with Rack::Lint. The
particular server configuration (which I in no way control), throws
500 errors on its own attempts to send 304: not modified. This
caused me to write a workaround for File::Download (sub
lintWorkAround here), so I'm not above putting blame on the
server in this instance also, if warranted. I don't believe I buggered
up the chunk-handling code from File::Download 0.3 (see diff),
but I suppose that's also possible. Is it possible to request a
particular chunk size from the server?
I'm using LWP and libwww versions 6.13 in perl 5.18.2.
File::Download version is my own 0.4_050601.
So, what else could the 406 error mean? Is there a way to request that
the server temporarily cache/version control the entire file so that I
can download a given ETag'd file once the transfer begins?
I am writing some Elisp code that downloads files using url-copy-file, and most of the time it works fine, but sometimes the contents of the file end up being the http headers, e.g. the downloaded file has the following contents:
HTTP/1.1 200 OK
Server: GitHub.com
Date: Thu, 14 Nov 2013 20:54:41 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked
Status: 200 OK
Strict-Transport-Security: max-age=31536000
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-UA-Compatible: chrome=1
Access-Control-Allow-Origin: https://render.github.com
Or sometimes the following is appended to the end of an otherwise correctly-downloaded file:
0
- Peer has closed the GnuTLS connection
However, when these things occur, the function seems to return just fine, so there is no way for me to verify that the file has really been downloaded. Is there any more reliable way to download a file in elisp (without shelling out to wget/curl/whatever)?
As recommended, I have reported this as a bug: lists.gnu.org/archive/html/bug-gnu-emacs/2013-11/msg00758.html
I tried to login to Facebook with cURL, but then I got this error:
HTTP/1.1 200 OK
Cache-Control: private, no-cache, no-store, must-revalidate
Expires: Sat, 01 Jan 2000 00:00:00 GMT
P3P: CP="Facebook does not have a P3P policy. Learn why here: http://fb.me/p3p"
Pragma: no-cache
Set-Cookie: datr=_KpbTjHjH8CWs_EOFbCp4H5r; expires=Wed, 28-Aug-2013 15:06:36 GMT; path=/; domain=.facebook.com; httponly
Set-Cookie: lsd=Mh7wl; path=/; domain=.facebook.com
Set-Cookie: reg_fb_gate=https%3A%2F%2Fwww.facebook.com%2Flogin.php%3Flogin_attempt%3D1; path=/; domain=.facebook.com
Set-Cookie: reg_fb_ref=https%3A%2F%2Fwww.facebook.com%2Flogin.php%3Flogin_attempt%3D1; path=/; domain=.facebook.com
Content-Type: text/html; charset=utf-8
X-FB-Server: 10.33.22.124
X-Cnection: close
Content-Length: 24058
Date: Mon, 29 Aug 2011 15:06:36 GMT
What's wrong?
It might be because you're not allowed to crawl the website nor access it with a robot. They have an API that can be accessed via cURL.
Using Chrome DevTools in your web-browser (also in Firefox, Opera and other), you can browse all pages and media files in Application tab. To convert it into curl command, go to Network tab (refresh page if necessary or tick Preserve log), find the page of your interest (list can be filtered by Img), right click on the request, then select Copy as cURL. Inspect the content in your clipboard or just paste the command to the terminal.
To convert from curl format to wget, do the following conversions:
remove --compress parameter,
change -H to --header in all places.
Then run the command.
Doing that, please consider Facebook's Automated Data Collection Terms.
If you're interested in a backup of all your data, check: Accessing & Downloading Your Information.
Related:
How to download files from Facebook group?
How to download Facebook page?
wget for fetching Facebook profile/friend pages
How do I use wget/curl to download from a site I am logged into?
How to download this webpage with Wget?
Using wget to download images in facebook conversations
When we visit fm.im, we are redirected to http://friendfeed.com.
Here are some other examples:
ff.im/abc
ff.im/efg
How is FriendFeed able to do this?
.im is the Isle of Man top-level domain (ccTLD). The registry normally requires names to be at least three characters long, unless you pay considerably more.
Two-character domains look cool but aren't particularly useful since IE rejects their cookies (old article, but still mostly true for newer IE versions).
When your browser requests ff.im:
GET / HTTP/1.1
host: ff.im
their webserver responds with a redirect, either to the main FriendFeed site:
HTTP/1.1 302 Found
Date: Sat, 09 Apr 2011 12:29:38 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Content-Length: 0
Location: http://friendfeed.com/
Server: FriendFeedServer/0.1
or to some other place (when using their URL-shortener).