I tried to login to Facebook with cURL, but then I got this error:
HTTP/1.1 200 OK
Cache-Control: private, no-cache, no-store, must-revalidate
Expires: Sat, 01 Jan 2000 00:00:00 GMT
P3P: CP="Facebook does not have a P3P policy. Learn why here: http://fb.me/p3p"
Pragma: no-cache
Set-Cookie: datr=_KpbTjHjH8CWs_EOFbCp4H5r; expires=Wed, 28-Aug-2013 15:06:36 GMT; path=/; domain=.facebook.com; httponly
Set-Cookie: lsd=Mh7wl; path=/; domain=.facebook.com
Set-Cookie: reg_fb_gate=https%3A%2F%2Fwww.facebook.com%2Flogin.php%3Flogin_attempt%3D1; path=/; domain=.facebook.com
Set-Cookie: reg_fb_ref=https%3A%2F%2Fwww.facebook.com%2Flogin.php%3Flogin_attempt%3D1; path=/; domain=.facebook.com
Content-Type: text/html; charset=utf-8
X-FB-Server: 10.33.22.124
X-Cnection: close
Content-Length: 24058
Date: Mon, 29 Aug 2011 15:06:36 GMT
What's wrong?
It might be because you're not allowed to crawl the website nor access it with a robot. They have an API that can be accessed via cURL.
Using Chrome DevTools in your web-browser (also in Firefox, Opera and other), you can browse all pages and media files in Application tab. To convert it into curl command, go to Network tab (refresh page if necessary or tick Preserve log), find the page of your interest (list can be filtered by Img), right click on the request, then select Copy as cURL. Inspect the content in your clipboard or just paste the command to the terminal.
To convert from curl format to wget, do the following conversions:
remove --compress parameter,
change -H to --header in all places.
Then run the command.
Doing that, please consider Facebook's Automated Data Collection Terms.
If you're interested in a backup of all your data, check: Accessing & Downloading Your Information.
Related:
How to download files from Facebook group?
How to download Facebook page?
wget for fetching Facebook profile/friend pages
How do I use wget/curl to download from a site I am logged into?
How to download this webpage with Wget?
Using wget to download images in facebook conversations
Related
My initial response headers - notice the Accept-Ranges header
HTTP/1.1 200 OK
X-Powered-By: Express
Vary: Origin
Access-Control-Allow-Credentials: true
X-RateLimit-Limit: 1
X-RateLimit-Remaining: 0
Date: Thu, 08 Apr 2021 06:14:19 GMT
X-RateLimit-Reset: 1617862461
Accept-Ranges: bytes
Content-Length: 100000000
Content-Type: text/plain; charset=utf-8
Content-Disposition: attachment; filename="some_file.txt"
Connection: keep-alive
Keep-Alive: timeout=5
I then restart the server and click resume download in chrome, but chrome doesn't send back in Range request headers
I'm following the documentation on Mozilla's website
Am I missing a header or misunderstanding how this works, especially with chrome and other browsers? Is there another way I can manually support resuming downloads by sending the right response and understanding the right request? From a technical perspective, if chrome sends back which range it now needs I will be able to resume the download.
According to this article, chrome should support something like this. I just need to be pointed in the right direction.
Ty!
Chrome needs some way to know that the file it's trying to download at that URL is indeed the same file when it tries to resume.
If you add support for an ETag header, this will likely work.
I am trying to upload a video to Twitter using the chunked upload endpoint. For this I post to the endpoint '**https://upload.twitter.com/1.1/media/upload.json**' the 'INIT' command. n posting to the Twitter server I get the error below
{"errors":[{"code":32,"message":"Could not authenticate you."}]}
At first I had a doubt regarding the Oauth signature generation function. But I can post a tweet and upload a .png image to the https://upload.twitter.com/1.1/media/upload.json using the Oauth signature generated by this Oauth signature generation function
I could also post the 'INIT' command initiate the chunked upload of a video successfully using TWURL
I have pasted below the actual request and response which was captured using Fiddler
Request
POST https://upload.twitter.com/1.1/media/upload.json HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: upload.twitter.com
Authorization: OAuth oauth_consumer_key="", oauth_nonce="MDAwMDAwMDAtMDAwMC0wMDAwLTAwMDAtMDAwMDAwMDAwMDAw", oauth_signature_method="HMAC-SHA1", oauth_signature="BtxgmRxA1bt5FI2Hu3qhhVIb5Eg%3D", oauth_timestamp="1607577817", oauth_token="****", oauth_version="1.0"
Content-Length: 357
Connection: Keep-Alive
command=INIT&media_category=tweetvideo&media_type=video%252Fmp4&oauth_consumer_key=******&oauth_nonce=MDAwMDAwMDAtMDAwMC0wMDAwLTAwMDAtMDAwMDAwMDAwMDAw&oauth_signature=BtxgmRxA1bt5FI2Hu3qhhVIb5Eg%3D&oauth_signature_method=HMAC-SHA1&oauth_timestamp=1607577817&oauth_token=********&oauth_version=1.0
Response
HTTP/1.1 401 Authorization Required
cache-control: no-cache, no-store, max-age=0
content-length: 64
content-type: application/json; charset=utf-8
date: Thu, 10 Dec 2020 05:23:39 GMT
server: tsa_k
set-cookie: personalization_id="v1_jQ8zK7e0TgY2uCQKkiVOgA=="; Max-Age=63072000; Expires=Sat, 10 Dec 2022 05:23:39 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None
set-cookie: guest_id=v1%3A160757781933882195; Max-Age=63072000; Expires=Sat, 10 Dec 2022 05:23:39 GMT; Path=/; Domain=.twitter.com; Secure; SameSite=None
strict-transport-security: max-age=631138519
vary: Origin
x-connection-hash: 0e6ebd96e051b8f7a413fb79c51a8f42
x-frame-options: SAMEORIGIN
x-response-time: 182
x-tsa-request-body-time: 0
x-xss-protection: 1; mode=block
{"errors":[{"code":32,"message":"Could not authenticate you."}]}
i googled a lot for a solution to this issue but none of the solutions are working. I also read the relevant Twitter documentation many many times
Thanks in advance for any tips
Mathew
Finally found what was causing the error. While passing the URL to which data has to be posted to the function that I use to generate the Oauth signature I was passing 'https://upload.twitter.com/1.1//media/update.json.com' instead of 'https://upload.twitter.com/1.1/media/update.json.com'. One additional '/' character was the reason why the error was occurring.
Thanks and regards
Mathew
When using the Structured Data Testing Tool to test my Mom's recipe site page titled Perfect Chicken Fajitas I get the following...
ERROR
The URL was not found. Make sure the domain name is correct and the server is responding with a 200 status code.
However, if I curl for the same URL, I can see that a 200 results...
$ curl -I http://www.lindysez.com/recipe/perfect-chicken-fajitas/
HTTP/1.1 200 OK
Content-Length: 0
Content-Type: text/html; charset=UTF-8
Server: Microsoft-IIS/7.5
Set-Cookie: bb2_screener_=1457484500+172.4.33.122; path=/
X-UA-Compatible: IE=edge
Link: <http://www.lindysez.com/wp-json/>; rel="https://api.w.org/"
X-Powered-By: ASP.NET
Date: Wed, 09 Mar 2016 00:48:21 GMT
What could be the problem?
I'm trying to access Microsoft's Outlook.com Calendar REST API. I got OAUTH2 authentication set up correctly and have a valid access token and refresh token available.
However, if I try to access the calendar list # https://outlook.office.com with my access token:
GET /api/v1.0/me/calendars HTTP/1.1
Accept: application/json; odata.metadata=none
Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciO[...]
the service returns with an 403 Forbidden
HTTP/1.1 403 Forbidden
Cache-Control: private
Transfer-Encoding: chunked
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Server: Microsoft-IIS/8.5
Set-Cookie: ClientId=OD6KHQBTKOKMLXUI8OJEG; expires=Wed, 21-Sep-2016 18:37:21 GMT; path=/; secure; HttpOnly
Set-Cookie: exchangecookie=8e4f582170cb445780c7148e9494b293; expires=Thu, 22-Sep-2016 18:37:23 GMT; path=/; HttpOnly
Set-Cookie: ClientId=OD6KHQBTKOKMLXUI8OJEG; expires=Wed, 21-Sep-2016 18:37:21 GMT; path=/; secure; HttpOnly
request-id: 7113f37d-69e0-4f8c-a264-9f3599d47899
X-CalculatedBETarget: CY1PR08MB1801.namprd08.prod.outlook.com
X-BackEndHttpStatus: 403
OData-Version: 4.0
X-AspNet-Version: 4.0.30319
X-DiagInfo: CY1PR08MB1801
X-BEServer: CY1PR08MB1801
X-Powered-By: ASP.NET
X-FEServer: AM3PR04CA0074
X-MSEdge-Ref: Ref A: D69A31E4FAA44258B0B8C351A71D2F9E Ref B: 0D3CA60C0976F50C452293F8CF403D8C Ref C: Tue Sep 22 11:37:23 2015 PST
Date: Tue, 22 Sep 2015 18:37:22 GMT
{"error":{"code":"ErrorAccessDenied","message":"Access to OData is disabled."}}
I played around with Outlook's OAuth Sandbox, where the same request returns a 200 OK with the correct data in the body. Strangely enough, they spot a nice little Show me the cURL! button in the sandbox, but this exact curl command will fail again with a 403 error in my terminal.
What am I missing?
I talked to a Microsoft techie. My problem was, that the REST API is not public (yet) and Microsoft has not yet enabled the API for standard Outlook.com accounts.
I requested an Outlook developer preview account with the REST API enabled by writing an email to outlookdev#microsoft.com
It took some time until I got a response from Microsoft, but using the developer account everything works now as expected.
When we visit fm.im, we are redirected to http://friendfeed.com.
Here are some other examples:
ff.im/abc
ff.im/efg
How is FriendFeed able to do this?
.im is the Isle of Man top-level domain (ccTLD). The registry normally requires names to be at least three characters long, unless you pay considerably more.
Two-character domains look cool but aren't particularly useful since IE rejects their cookies (old article, but still mostly true for newer IE versions).
When your browser requests ff.im:
GET / HTTP/1.1
host: ff.im
their webserver responds with a redirect, either to the main FriendFeed site:
HTTP/1.1 302 Found
Date: Sat, 09 Apr 2011 12:29:38 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Content-Length: 0
Location: http://friendfeed.com/
Server: FriendFeedServer/0.1
or to some other place (when using their URL-shortener).