IS there a limit of connections and time of requests I can send to the Facebook public API?
I do not need to get private informations of the logged in users, but only some public information.
I will get a LOT of data, and I will have to do a LOT of request.
Is there a limit I have to consider?
There is officially no limit that is specified by the Facebook Dev team. However, users did face issues in the past while trying to pull large amounts of data. I believe there was also a bug that was being tracked some time back regarding this.
Facebook limits API access on multiple levels as e.g. per User ID and per App ID. The calculate a 'score‘ per app that defines individual API limits. This depends e.g. on the number of active users but also on the 'smartness' of the requests as well as their CPU and memory usage on the Facebook side. See this for more details https://developers.facebook.com/docs/reference/ads-api/api-rate-limiting/
Related
Background
Given a symbol (i.e. cashtag), I need to retrieve twits from 2015 up to 2019 in order to perform sentiment analysis for a research project.
Problem
The public API is limited to the last 30 messages, as explained in https://api.stocktwits.com/developers/docs/api#streams-symbols-docs.
https://api.stocktwits.com/api/2/streams/symbol/NVDA.json
I tried the same request in an authenticated manner, but I face the same limitation.
https://api.stocktwits.com/api/2/streams/symbol/NVDA.json?access_token=<access_token>
Question
Is there a way to work around the limitation ?
Yes, there is. You should contact their support team, to get Partner level access to download historical data.
"There are some endpoints and data that are not available in the public Stocktwits API, including the full firehose data stream of our content. You may also have an application that requires higher usage limits (rate, message or other limits) than the public API. These additional capabilities are highlighted in the API documentation. If your application requires access to these capabilities, please contact our team to discuss."
https://api.stocktwits.com/developers/contact
For my application/crawlers I collect lots of data which leads to exceeding the rate limit very often. I crawl News-Pages and stuff like that so the Token doesn´t need any permissions like posting stuff.
When using the Graph Explorer you can create an User Access Token(lasts 1-2 hours before expiring). You can create as many as you want so I thought it may be possible to abuse this and overcome the rate limit. I tested it and it somehow worked. Did like 6000 API calls with 2 Tokens in under an hour.
Questions:
Did someone else try this already? If so, did Facebook notice and shut down the Account?
Is it possible to request a new User Token from the Graph Explorer via Code or something else like a virtual machine running with an mouse makro to generate new Tokens every ~30mins?
Yes. Yes :) It can go til banning the account or the IP from which the requests are made.
Access Tokens can be obtained by code and you can create more and make some balancing between them combined with different proxies you request through.
HOWEVER I recommend you to use the Facebook ways and respect their politics.
Does anyone know if there is a way to increase the quota limit of 10 queries when batching calls to the core reporting API?
This question/answer mentions the limit of 10: How can I combine/speed up multiple API calls to improve performance?
If I try to add more than 10 queries to the batch only the first ten are processed, each one after that contains a 403 quota exceeded error.
Is there a pay option? Would love to speed up the process of reporting on GA data for a bunch of URLs. I looked in my Google Developer's Console under the Analytics API where there is an option to increase the per-user limit and a link to request additional quota but I don't need total quota to increase, only allowed batch requests.
Thanks!
Quota is the number of requests you are allowed to make to a Google API without requesting permission to access more. Most of the Google APIs have a free quota, a number of requests Google lets you make without asking for permission to make more request. There are project based quotas and user based quotas.
Unless it says other wise APIs Quotas are projects based not user based.
User quota example
Per-user limit 10 requests/second/user
Some Quotas are user based, a user is normally the person that has authenticated the request. Every request sent to google contains information about who is making the request in the form of the IP address where the request came from. If you have your code running on a server the IP address is the same all the time so Google sees it as the same user. You can get around his by adding a random Quotauser to your request this will identify the request based upon different users.
If you send to many requests to fast from the same user you will see the following error.
userRateLimitExceeded The request failed because a per-user rate limit
has been reached.
The best way to get around this is to use QuotaUser in all of your requests, and identify different users to Google. Or just send a random number every time should also work.
Answer: You can't apply for an extension of the flood protection user rate limit. But you can get around it by using QuotaUser.
more info on quotas can be found on Google developers console APIs
So I'm struggling to find where this is documented (if at all), but I'm getting the following error message when requesting data from the FB GraphAPI.
"Please reduce the amount of data you're asking for, then retry your request"
The call I'm making is:
/v2.3/user1/posts?fields=object_id&limit=100
If I change it to:
/v2.3/user2/posts?fields=object_id&limit=100
It returns 100 items.
Why would it work for one user, and not the other?
Both requests are authenticated via an access token (not belonging to either user) and I get the same error whether running it from my code, or the Facebook Graph API console of developers.facebook.com
The response from CBroe is correct. Facebook returns this error if it finds that too many internal resources are needed to respond to your request.
Therefore you have to do what the response says: limit it.
This can be done in (afaik) 2 ways:
Use the limit parameter and reduce the amount of responses you expect from the API
Provide a timeframe (using since and / or until) to fetch only data (posts / videos) for a specific timeframe.
We had the same issue as you, but with retrieving videos from a page. Unfortunately using the limit parameter did not work, even when I set it to limit=1. But by using the since / until parameters we finally got results.
Therefore I suggest to implement a timeframe in order to reduce the amount of data, or alternatively, split the amount of requests you make. e.g. if you want all posts from the past 3 months and run into the mentioned error: split your requests in half using since and until. If that still does not work: keep splitting...
=> Divide and conquer ;)
Hope it helps,
KR, ebbmo
Recent bug filed on FB talks about the same error. They seem to accept that this could be a bug, but not much other information forthcoming.
https://developers.facebook.com/bugs/1904674066421189/
There are both app-level and user-level rate limits that are enforced on Graph API calls. In your case, it could be that you've made a large number of calls in a short time with user1.
You can check out this page for more about Facebook's rate limits: https://developers.facebook.com/docs/marketing-api/api-rate-limiting (even though the URL refers to the Marketing API, the information also applies to the Graph API.)
I have tried raising this concern on Facebook/Support/Bugs but they said I should post implementation issues here. I have read it everywhere and it seems to be quiet open issue till now. I am not sure, If this will be solved or not.
So, what we are doing is, we have clients - Android and iOS.
Apps on Android/iOS allows users to login into the app and generate the token on the basis of permissions set we have, and we are passing this token to server for fetching further data as and when required for client. As our userbase is increasing we are getting Application request limit reached quiet often.
We are fetching photos of users and their friends using FQL. So, when parallely fetching photos for around 8-10 different users, we are reaching the Application request limit sometimes, which is quiet random and we are not aware of the actual scenario when it breaks up and how. According to facebook the limit, which is 1M calls per day, but we are hitting around 80K - 1 Lac API calls in a day, but as users are increasing it is stretching a bit further, Less than or equal to 200 approax calls/user. We tried doing batch calls as well and we hit the application request limit as well.
If anyone of you could help us understand the complete concept of API limit and how this can be handled, then we will really appreciate the help. We want to understand how API limit is decided and it's rate is calculated over which interval so that we will be able to configure on our side accordingly.
Earlier in the day, we ran into a unique API call issue. Our server started to break for API calls for user tokens that are with us, we (on our systems, other than server) tried fetching the data for those tokens (Simple calls - /me or /me/home), and it was working alright for us but not for server, then we tried setting up another server and redirected the requests to our new server then this server works well for the same set of users. Not sure, what went wrong in this case and how it breaks up. Please help.
Many Thanks,
Reno Jones
Did you look at the Insights -> Developer section of developer.facebook.com for your app?
This will show you a breakdown per api call, including warnings and ones that are currently being throttled and why.
Also, are you sure you're using User token authorization and not just your App token?
Beyond that, we take the information from Insights to find api calls to cache on our side rather than hitting Facebook every time. You will likely have to do something similar if you're not already. They have limits for calling too often, as well as for requesting too much data. For those, we had to reduce the limits of historical data we requested.