Amazon has an administration page for content sent to your Kindle. This page uses an undocumented HTTP API that sends requests like this:
{
"csrfToken":"gEABCzVR2QsRk3F2QVkLcdKuQzYCPcpGkFNte0SAAAAAJAAAAAFkUgW5yYXcAAAAA",
"data":{"param":{"DeleteContent":{"asinDetails":{"3RSCWFGCUIZ3LD2EEROJUI6M5X63RAE2":{"category":"KindlePDoc"},"375SVWE22FINQY3FZNGIIDRBZISBGJTD":{"category":"KindlePDoc"},"4KMPV2CIWUACT4QHQPETLHCVTWEJIM4N":{"category":"KindlePDoc"}}}}}
}
I made a wrapper library for the previous API they used, but this time they have added CSRF tokens, making each session unique. That is a bit of a show stopper, and I was wondering how I can get hold of these tokens. I did not find it in the cookies. This is for use in a Chrome Extension, so issues like CORS is not an issue.
Actually, after manually going searching the Response tab of each request in the "XHR" and "Doc" tab, I was able to find out that this token is set in an inline script in the myx.html (main page):
var csrfToken = "gPNABCIemSqEWBeXae3l1CqMPESRa4bXBq0W7rCIAAAAJAAAAAFkUlo1yYXcAAAAA";
This means it is set on the window object, making it available for all there. I guess this means a Chrome extension would need to fetch this page and manually parse the html to retrieve this token. Sad, but doable, although highly fragile :-(
Related
How can I use personal access token to view raw content of files in GitHub? I tried to do it with token#raw..., username:token#raw... etc. but none of them worked.
IMPORTANT NOTE: I need to use the token in URL.
This format (replacing the capitalized sections) should work:
https://TOKEN#raw.githubusercontent.com/USER/REPO/BRANCH/PATH
Note that this appears to block web browsers, you'd have to make a GET request from an HTTP client (like Insomnia) to test it. If this doesn't work in your plugin, please explain what happens, as it may be sending the wrong headers.
I need to scrape a webpage.
I specifically need to extract the section "USDT Funding Market" on the page.
$WebExtract = Invoke-WebRequest -Uri "https://www.kucoin.com/margin/lend/USDT"
Although $WebExtract.content or $WebExtract.AllElements does not contain the "USDT Funding Market".
The reason for this is that your initial web request is only returning you the HTML of the page, however a lot of modern web applications like this have some JavaScript under the hood that will reach out to their API's and fill the page with data after the initial HTML is loaded.
The good news is, usually with public websites like this you can hit the underlying API's without worrying about authentication - which actually makes parsing the data easier, as it's typically in JSON format.
Try opening the debug console in your browser (usually F12), open the network tab and refresh the page - you can typically find these API requests/endpoints in there, e.g;
Simply copy that URL out and invoke the web request to the API instead, for example;
$request = Invoke-WebRequest -Uri 'https://www.kucoin.com/_api/currency/prices?base=USD&targets='
$json = $request.Content | ConvertFrom-Json
$json.data
This might not be the specific API endpoint/data you're after, but hopefully this points you in the right direction. If you poke around in the console you should be able to find the one you're looking for.
EDIT:
Apologies for the misleading information here, I didn't notice that these numbers were updating in real time on the site. The above is still pretty common for a lot of websites so I'll leave it, however on sites that are this responsive/live another common technology used is Web Sockets.
If you have a look at the last lines of the console, you should be able to see a request ending with 'transport=websocket' to a URL like this;
wss://ws-web.kucoin.com/socket.io/?token=...%3D%3D&format=json&acceptUserMessage=false&connectId=connect_welcome&EIO=3&transport=websocket
If you select this line and head over to the Messages tab in your browser console/debugger, you'll be able to see the web socket messages being returned;
This looks like the data you're after, but I'm not hugely familiar with querying Web Sockets through PowerShell.
Simply invoking a web request won't work here, as it's using the web sockets protocol to handle the communication. You would also need to find a way to get a valid token for opening a web sockets connection (likely just a web request for that).
Perhaps these posts will help;
http://wragg.io/powershell-slack-bot-using-the-real-time-messaging-api/
How to use a Websocket client to open a long-lived connection to a URL, using PowerShell V2?
So when using Facebook login, it turns out that if you set response_type to token or code%20token then the "response data is included as a URL fragment" on the URL that redirects to your app. Seriously, why? When would that ever be useful to anyone?
Anyway, I'm hoping that this is somehow security related and that it's there for a reason other than to annoy people, but I would otherwise like to know if there's a way to disable it, since I'm using Angular and it's a real pain to handle urls with hashbangs.
If I set a response_type of code only, the code is returned as a nice and clean query parameter, but I would like to receive both in order to perform additional validations. Is there a way to change this behaviour? Thanks.
EDIT:
Well it turns out that setting a response_type of code returns #/= at the end of the url, so there's no way to get a clean querystring. It has already been established that there is no way to change this behaviour, but I'm still interested in finding out why Facebook is doing it. Is it security related? What is the purpose of these url fragments?
Answers
To your first question, I imagine you would use token when you are handling everything in-browser and not processing at the server.
To your main question, as far as I know, you can not change how Facebook redirects successful authorizations. Facebook doesn't give you the token as a clean query parameter. If you use token or code token, what you want will be in the hash fragment. Sorry.
NOTE: This may not be a problem for you. I'm not sure what you want with "additional validations", but when you use code token, exchanging the resulting code gets you a different token than the one you just got embedded in the URI. Both are now valid and will expire separately. Really, you probably need either the code or the token; both won't help you since they're not linked.
Examples of using all three methods:
If you set response_type to code you get redirected to something like:
http://example.com/redirect_uri?state=thestatepassed&code=AQDN9E9GYjA8NbyCt
87_jV5vHnCQylNxmBswo6Z1BsrR7lmTPom6wjrzfan6P4GBLDt3EQrfPg0xSLoMLxBBfscsyfSY
JNM2vu9OoqEQXXSJCTUq_fMpUwqkYbCHp-GAqL4H1ymbMz7zPKAG61V9BtKTSuez39yhawOu7l-
6ww4thP41Ka9PVcknTQ6fPjPXKYSyxEmANps9zevCPFsXpBZCO7_dms65-ZZuG2wVBd16gFnBZH
q8EY0qih6-9o61wXh7bBvVPVSZ2im7Oj1nx47YgDpbD3X0XdlVhUoGYmBdER9hNmIC2PmmY7VAo
PlYCZc#_-_
From there, you need to exchange the code through Facebook's OAuth endpoint to receive an access token. The access token can then be used against the Facebook APIs.
If you set response_type to token you get redirected to something like:
http://example.com/redirect_uri#access_token=CAACYnSxGEhsBAJBg0ohZBhAf7pKEU
sm5ytZAZBzKjISFuRun2ZByZCqEsxrVIgtiO7iIlJZBBbGm6fRPQXItZCX6YgjPknUBsr78tJtv
W6fySULrUo9vdW57ZCMUUIlNaeZAcU8DzUXKmFpgotOyhE3jXYz1c3eu00Aii0AZBsPrtrwjpwQ
mV8VYQNiqKZBIsqOrIwZD&expires_in=4168&state=thestatepassed
You have everything you need to call the Facebook APIs. The access token returned is valid, but should probably be checked against Facebook's token inspection endpoint if you're doing something server-side. (Really, at that point, just use code. I've never done it this way, so good luck.)
As you noted, the access token is now contained within the hash fragment of the url.
If you set response_type to code token you get redirected to something like:
http://example.com/redirect_uri#code=AQAtzsjPivFPsJ538KFlPuhLaK6pDMlrGDiwmi
KDcpgNfWrO1EdX5i6zK_Op2D0QDEXZLyifXxh4TSeBZCWhnkl7YV1LMyEkbPURAWSoqRoeG7tfM
4nB4nDAHOK0H9umb0KnoypRT1pP05FJKhl2QjpCJrPPFDHl6y-1X9ZMj1uVHtmPNi4tG_6QAbuL
RaGadBkekb22uJ0iwSrWc9OKi6ET70lCTYb18hbwUkzHXtTq12nNEdsDJ7Ku2wEBwMygFwErYDX
CrnPoFoah_z0UPCfv3XZLy98Dhlzw_lnx8nnCB-PCppOWRqmydvQJehPd86k&access_token=C
AACYnSxGEhsBALXHRQwfm4UoauRlZBJDVpZCiM6ZCuM3bE965F5JVBfBB8inTFdhfJ5obnonSqa
m3v8FbWhHXrhRSx4ugwAmmDaWyxmPELWqSrkrDO5ueTUXhhjiEZBTd7HjCVCSOXXhOSo3DjEVSC
lOaZBfqmXsprYyc6LJC39sroCcHYCZCv&expires_in=5183426&state=thestatepassed
Now you have both an access token and a code (that expiration applies to the token, not the code). As stated earlier, that code can be exchanged for an access token in the usual way, but the returned access token will be different from the one you just got embedded in the URI.
But, again, what you care about is in the hash fragment.
See the AngularJS doc on $location for accessing the hash. Someone more knowledgable than me can speak to how that works with routes and how best to scrape the parameters.
Well today I had the same situation and managed to resolve it in a way I could get the access token from my server.
After getting the code using response_type=code I called via GET the following url:
https://graph.facebook.com/v3.2/oauth/access_token?app_id=MY_APP_ID&code=MY_CODE&client_id=MY_APP_ID&client_secret=MY_APP_SECRET&redirect_uri=YOUR_LOGIN_REDIRECT_URI
This route returns a JSON response, which has the access_token parameter, in this way:
{
"access_token": "ABAECAEFAEFAEA...",
"type": "bearer",
"expires_in": 838383838
}
So there you go :) I hope this works for you too
I'm using FB.login on the JS client and want to verify the user's identity on the server. So, the client gets a signedRequest from facebook and sends it to the server. The server splits on the period, and decodes the second part of the signedRequest into a json object.
What should I be using for "code" when I send my server-side request to
https://graph.facebook.com/oauth/access_token?
client_id=YOUR_APP_ID
&redirect_uri=YOUR_REDIRECT_URI
&client_secret=YOUR_APP_SECRET
&code=CODE_GENERATED_BY_FACEBOOK
My decoded json looks something like:
{"algorithm":"HMAC-SHA256","code":"2.AQCPA_yfx4JHpufjP.3600.1335646800.1-5702286|l11asGeDQTMo3MrMx3SC0PksALj6g","issued_at":1335642445,"user_id":"5232286"}
Is that the code I need? Does it need to be B64 encoded? If this isn't the code, what code should I use?
_
What I've tried:
The request I'm trying to use is:
https://graph.facebook.com/oauth/access_token?client_id=295410083869479&redirect_uri=https://squaredme.appspot.com/facebookredirect&client_secret=44f1TOPSECRETbb8e&code=2.AQCPA_yfx4JHpufjP.3600.1335646800.1-5702286|l11asGeDQTMo3MrMx3SC0PksALj6g
but this returns the error:
{"error":{"message":"Error validating verification code.","type":"OAuthException","code":100}}
I can't tell if this is because I'm using a bad code, or what. Noteably, this is running on my local dev server, and squaredme.appspot.com definitely does NOT resolve to my IP. I don't know if facebook checks that or what - I'm assuming I'd get a better error message. Thanks for any direction!
You are trying to somehow combine the two flows together and that's why things don't work well.
When facebook POSTs into the iframe with your app url and a signed request there are two options, the easy one being that the user is already authenticated and then the signed request will have all the necessary data (including a signed request), then you just load the canvas page and use the JS SDK to get an access token there as well, but in this case there's no need to use the FB.login (since it opens a popup and will automatically close it), you can use the FB.getLoginStatus method which won't annoy the user.
If the user is not authenticated then the sign request will be missing the things you need to use the graph api.
You then redirect the user to the auth dialog, and since you are loaded in an iframe you'll need to return a html response which redirects the parent window using javascript, like:
top.location.href = "AUTH_DIALOG_URL";
When the use is done (accepted or rejected the app) he will be redirected to the "redirect_uri" you added as a parameter to the auth dialog.
If the user accepted your app then you'll be getting the "code" parameter in the query string.
You then take the code, exchange it with an access token as you posted in your question, and then redirect the user back to "apps.facebook.com/YOUR_APP".
When the page then loads the user is already authenticated and you'll be getting a full signed request.
I hope this clarifies things for you, recheck the Server-Side flow it pretty much covers it all.
I also had some trouble with that, then I found the solution here in StackOverflow.
There are two kinds of "code" provided by facebook. One comes inside the signedRequest in the cookie generated by the client-side flow. The Facebook's JS SDK handles this codes and get a access token without telling us anything.
The other type of code comes attached as a query to your redirect URI (http://www.yoururl.com/index.php?code=AAAgyiaus...), when you navigate to OAuth URL (server-side flow). With this code, you go to a Token URL and get your access token.
When you are using the server-side flow, you need to indicate a redirect URI both in the OAuth URL AND in the Token URL, and they have to be exactly the same, so a missing slash or a query string can be a lot of problem.
The codes are different from each other. When you use the both things together, appears to be impossible to get a access token using the code that was inside the cookie's signedRequest.
BUT, it is not. The magic is: the code from signedRequest is associated with NO URI, so as long as the redirect_uri is a mandatory field, all you have to do is to pass it blank when you navigate to the Token URL.
So the final solution is: grab the signedRequest from the cookie, parse it in your server to obtain the code, then read the Token URL:
https://graph.facebook.com/oauth/access_token?
client_id=YOUR_APP_ID
&redirect_uri=&client_secret=YOUR_APP_SECRET
&code=CODE_INSIDE_THE_SIGNED_REQUEST
It looks like a hack, so I don't know how long it's gonna work, but it's working right now.
Is there a way to go to a url without redirecting to it? Basically I want to call a url from within my application in the background so it can logout a reliant party.
Appreciate the help.
What you are trying to do does not compete us to answer as it's directly related to your own Authentication implementation.
A normal ASP.NET Authentication based in Forms Authentication you will need always to lunch the url from a browser as it is there that relies the Authentication given.
You can give yourself a try by opening your website and log in into it, after that, open other browser brand (not browser window) into your application url... you will see that you also need to login again as the Authentication is hook up into the first browser.
It's Up to you as Application Architect to make this by implementing another way of authentication, normally in this kind'a cases, this happend when consuming web services where you need a authentication code first (given by calling a Login method) and that code is always needed to be appended to the body or header of any call to the system.
This way you can easily remove the authentication code and all procedure calls will fail.
As said, this is not up to us, it's up to you to create the correct Authentication Layer.
from your comment
it's as simple as using WebClient object
WebClient client = new WebClient ();
string reply = client.DownloadString (address);
If you wish to transfer to a new url request you can still use
Server.TransferRequest()
The problem with this is that by not using a redirect the browsers address bar will not reflect the fact that you have moved their request to another URL.
To have the client visit a given URL in the background you should either make an AJAX call to it or possibly have an image with an src of your logout url (though you'd have to make sure that you return a FileResult of your image too). This is how most analytics packages call to their relevant urls in the background.
The problem here though is that neither is 100% reliable, turn off javascript or images on your browser and these results fail.
From what you've said I think what you're after is for a user to continue to any of a variety of pages rather than a specific logout page. If this is indeed the case your best solution is in fact a double redirect.
Have your application redirect to your logout url but before hand put the url of the page you want them to go to into tempdata. Then in the actionresult for the logout page you can do your logging out as required and return a redirect to the url from tempdata.