Although asked multiple times by users, I still feel uncomfortable with the theory behind.
I am using mongodb as a database for storing certain activities created by users (authenticated via facebook). For efficiency reasons, I want to store their profile picture URL along with the event (de-normalization in a event collection).
When the events are fetched from the database (by random visitors of my website), the profile pictures are rendered client side via:
<img src="https://graph.facebook.com/USERID/picture">
no access-token (due to security on the client side)
preferably lazy loaded...
For my understanding, the rate limit would become a problem, when I would do multiple request to the same profile picture server-side without passing an access token. (same request from same IP address)
Does the rate limit count against IP addresses? If so, only website visitors that
overrun the rate limit would have a problem whereas the same picture would still render just fine for other website visitors (different IP address) - is this correct?
To sum up:
server side requests to the graph API should be made with an access token (this applies the rate to an individual user... necessary because 1 server IP)
client side requests are made from different IP addresses anyway... so no access token required since every client (different IP address) counts against the individual limit
Are these assumptions correct?
If you absolutely wanted to nab those profile pictures you could do so client side as well. You could use a javascript file uploader and upload an off-screen facebook profile picture to your servers. Each user of your application could do some of the uploading for you.
Probably a huge violation of someone's terms of service, but an interesting thought experiment.
Related
I am writing a REST Api gateway for an Angular SPA and I am confronted with the problem of securing the data exposed by the API for the SPA against "data thiefs". I am aware that I can't do much against HTML scraping, but at least I don't want to offer such data thiefs the user experience and full power of our JSON sent to the SPA.
The difference between most "tutorials" and threads about this topic is that I am exposing this data to a public website (which means no user authentication required) which offers valuable statistics about a video game.
My initial idea on how to protect the Rest API for SPA:
Using JWTs everywhere. When a visitor opens the website the very first time the SPA requests a JWT from my REST Api and saves it in the HTTPS cookies. For all requests the SPA has to use the JWT to get a response.
Problems with that approach
The data thief could simply request the oauth token from our endpoint as well. I have no chance to verify that the token has actually been requested from my SPA or from the data thief?
Even if I solved that the attacker could read the saved JWT from the HTTPS cookies and use it in his own application. Sure I could add time expiration for the JWT
My question:
I am under the impression that this is a common problem and therefore I am wondering if there are any good solutions to protect against others than the SPA having direct access to my REST Api responses?
From the API's point of view, your SPA is in no way different than any other client. You obviously can't include a secret in the SPA as it is sent to anybody and cannot be protected. Also the requests it makes to the API can be easily sniffed and copied by another client.
So in short, as diacussed many times here, you can't authenticate the client application. Anybody can create a different client if they want.
One thing you can actually do is checking the referer/origin of requests. If a client is running in a browser, thr requests it can make are somewhat limited, and one such limitation is the referer and origin headers, which are always controlled by the browser, and not javascript. So you can actually make sure that if (and only if!) the client is running in an unmodified browser, it is downloaded from your domain. This is the default in browsers btw, so if you are not sending CORS headers, you already did this (browsers do, actually). However, this does not keep an attacker from building and running a non-browser client and fake any referer or origin he likes, or just disregard the same origin policy.
Another thing you could do is changing the API regularly just enough to stop rogue clients from working (and changing your client at the same time ofc). Obviously this is not secure at all, but can be annoying enough for an attacker. If downloading all your data once is a concern, this again doesn't help at all.
Some real things you should consider though are:
Does anybody actually want to download your data? How much is it worth? Most of the times nobody wants to create a different client, and nobody is that much interested in the data.
If it is that interesting, you should implement user authentication at the very least, and cover the remaining risk either via points below and/or in your contracts legally.
You could implement throttling to not allow bulk downloading. For example if the typical user accesses 1 record every 5 seconds, and 10 altogether, you can build rules based on the client IP for example to reasonably limit user access. Note though that rate limiting must be based on a parameter the client can't modify arbitrarily, and without authentication, that's pretty much the client IP only, and you will face issues with users behind a NAT (ie. corporate networks for example).
Similarly, you can implement monitoring to discover if somebody is downloading more data than it would be normal or necessary. However, without user authentication, your only option will be to ban the client IP. So again it comes down to knowing who the user is, ie. authentication.
I have a WordPress webpage with posts retrieving from a public Facebook page. The FB page is not mine. However the problem is that I have millions of visitors on my Web page and every time a user visits the web page it make an API call to FB Page. Since facebook allows only a limited number of API calls in a time frame, My limit reaches instantly with such a huge number of visitors. Is there any solution to this problem. an idea in my mind is:
1. retrieve posts from Facebook and store them locally and display them every time a user visits. Is it possible? If Yes where to start ?
Or can we get more API calls by paying facebook or things like that. I am ready to pay as far as my API calls can be made sufficient for my needs.
I am open to any solution and would be very thankful for any help to resolve the problem.
There are several possible solutions to this problem
Storing responses in database
You can add a middlepoint to your requests to Facebook API using your application. This would mean that you would have a database table which stores Facebook-related information, possibly along with a lifecycle time, like:
facebook_data(user_id, lifecycle_time, ...)
Whenever you would theoretically need to send a request to Facebook API, you can check the database table to see whether the user already has a record in that table and whether it is still valid. If so, give this data to the user. If not, send an API request to Facebook and store the response in this table.
Storing responses in localStorage/memory/file
You can also store Facebook-related data in the localStorage of the web browser of the memory of an app or a file, or even a local database specific for each user. This would prevent a lot of communication and server load your app is issuing on your server(s).
Queueing user requests to be sent
If the Facebook-related data is not very urgent to your users, you can queue requests to be sent and send a single request instead of a request for each user's each visit. You can do this via a cron job.
Periodically sending requests to Facebook
You can group your users into batches and periodically update their values via Facebook and storing in your database.
Combination
Naturally, you can combine the approaches, for instance, you can store in local memory, file, or localStorage values and in the database in the same time, so first locally stored information is searched for, not needing even a request if it exists and is still valid. If not, then checking the database record and using that if it exists and is still valid. And if the data is not found in local resources, nor your database, then you can send an API request.
For my application/crawlers I collect lots of data which leads to exceeding the rate limit very often. I crawl News-Pages and stuff like that so the Token doesn´t need any permissions like posting stuff.
When using the Graph Explorer you can create an User Access Token(lasts 1-2 hours before expiring). You can create as many as you want so I thought it may be possible to abuse this and overcome the rate limit. I tested it and it somehow worked. Did like 6000 API calls with 2 Tokens in under an hour.
Questions:
Did someone else try this already? If so, did Facebook notice and shut down the Account?
Is it possible to request a new User Token from the Graph Explorer via Code or something else like a virtual machine running with an mouse makro to generate new Tokens every ~30mins?
Yes. Yes :) It can go til banning the account or the IP from which the requests are made.
Access Tokens can be obtained by code and you can create more and make some balancing between them combined with different proxies you request through.
HOWEVER I recommend you to use the Facebook ways and respect their politics.
I am attempting to make a website's back-end API (I want to make the back-end independent of the front-end so I'm only making a server-side API for now, abiding to RESTfulness as much as possible). I haven't done this before so I'm unaware of the 'best' & most secure way to do things.
How I do it now:
Some parts of the API should only be accessible to a specific user after they login and up to 24 hours later.
To do this, I am generating a random Session ID whenever a user logs in (I'm using passwordless logins so the user is assigned that ID when they click on a link in their email) on the server side, which respond by sending that session ID to the client once. The client then stores this session ID in localstorage (or a file in disk if the client is not a web browser).
Next, I store that ID along with the associated email in my DB (MySQL table) on the server side.
Now every time the client want something from my API, they have to provide the email & session ID in the URL (I don't want cookies for now), which the server checks against the ones in the DB, if they exist then the server responds fully else responds with an error.
After 24 hours, the server deletes the email/session ID pair and the user has to login again (to generate another session ID and associate it with their email).
Now the questions:
Is my method secure or does it have obvious vulnerabilities? Is
there another battle-tested way I'm not aware of?
Is there a better way for the client to store the session ID (if
they are a web browser)?
What is the best way to generate a unique session ID? Currently I
generate a random 16-char string that I set as the primary key of
the session-email table.
Is using a MySQL table the most performant/best way to store session
IDs (given it will be queried with each request)?
Do I need to encrypt session IDs in any way? Is it secure for the
client to send it as a 'naked' URL param?
Sorry for having too many questions in one post but I think they're related by the single scenario above. If it makes any difference, I'm using F# and I expect my client to either be an android app or a web app.
Your REST API MUST not know anything about the REST client session, not even the session id. If you don't want to send a password by every request, all you can do is signing the user id, and the timeout, so the service can authenticate based on the signature. Use JSON web token: https://en.wikipedia.org/wiki/JSON_Web_Token
You can have a server side REST client, which can have the session your described. The question is, does it really worth the effort to develop a REST service instead of a regular web application? I am not sure in your case, but typically the answer is no, because you won't have any 3rd party REST client and your application does not have enough traffic to justify the layered architecture or it is not big enough to split into multiple processes, etc...
If security is important then you MUST use a true random generator algorithm or hardware. https://en.wikipedia.org/wiki/Random_number_generation#.22True.22_vs._pseudo-random_numbers It is not safe to send anything through HTTP, you must use HTTPS instead. You MUST use the standard Authorization header instead of a query param. https://en.wikipedia.org/wiki/Basic_access_authentication
I have tried raising this concern on Facebook/Support/Bugs but they said I should post implementation issues here. I have read it everywhere and it seems to be quiet open issue till now. I am not sure, If this will be solved or not.
So, what we are doing is, we have clients - Android and iOS.
Apps on Android/iOS allows users to login into the app and generate the token on the basis of permissions set we have, and we are passing this token to server for fetching further data as and when required for client. As our userbase is increasing we are getting Application request limit reached quiet often.
We are fetching photos of users and their friends using FQL. So, when parallely fetching photos for around 8-10 different users, we are reaching the Application request limit sometimes, which is quiet random and we are not aware of the actual scenario when it breaks up and how. According to facebook the limit, which is 1M calls per day, but we are hitting around 80K - 1 Lac API calls in a day, but as users are increasing it is stretching a bit further, Less than or equal to 200 approax calls/user. We tried doing batch calls as well and we hit the application request limit as well.
If anyone of you could help us understand the complete concept of API limit and how this can be handled, then we will really appreciate the help. We want to understand how API limit is decided and it's rate is calculated over which interval so that we will be able to configure on our side accordingly.
Earlier in the day, we ran into a unique API call issue. Our server started to break for API calls for user tokens that are with us, we (on our systems, other than server) tried fetching the data for those tokens (Simple calls - /me or /me/home), and it was working alright for us but not for server, then we tried setting up another server and redirected the requests to our new server then this server works well for the same set of users. Not sure, what went wrong in this case and how it breaks up. Please help.
Many Thanks,
Reno Jones
Did you look at the Insights -> Developer section of developer.facebook.com for your app?
This will show you a breakdown per api call, including warnings and ones that are currently being throttled and why.
Also, are you sure you're using User token authorization and not just your App token?
Beyond that, we take the information from Insights to find api calls to cache on our side rather than hitting Facebook every time. You will likely have to do something similar if you're not already. They have limits for calling too often, as well as for requesting too much data. For those, we had to reduce the limits of historical data we requested.