I'm a graduate student whose research is complex network. I am working on a project that involves analyzing connections between Facebook users. Is it possible to write a crawler for Facebook based on user's post information?
It is an unusual question, but..
I'm looking for someone who can do it together (if necessary even paid).
Building a Facebook crawler is against the TOS of Facebook:
https://www.facebook.com/legal/terms (3.2)
You will not collect users' content or information, or otherwise access Facebook, using automated means (such as harvesting bots, robots, spiders, or scrapers) without our prior permission.
Related
I have a question about Graph Api from Facebook. We want to create tool to management accounts in facebook and groups releated with them.
Basic functionality of tool is possibility to add acounts to the system, and then add you can add multiple facebook accounts to each of them. Then, system get all posts from groups releated with facebook accounts.
I did research, and I know quite what is possible and what no, but many endpoints of API requires app review. My question is about this review - wheater app like that, when one user can add multiple facebook accouns and use data (like posts, comments) from them is allowed by facebook and will pass the review?
That depends on what exactly you mean by “accounts” …
Facebook pages? Sure, you can let your app user add as many of those as they like, for the purpose of managing them, reading feed data, etc.
User profiles? That would not really make sense, because every person is only allowed to have one personal profile on the platform to begin with. And adding other people’s profiles (by having them authorize your app, using their access tokens on behalf of someone else) is likely not going to fly in review either.
when one user can add multiple facebook accouns and use data (like posts, comments) from them is allowed by facebook
“Using data” is rather vague as well; what you can and can’t do with user data is outlined in the Platform Policy: https://developers.facebook.com/policy
I know pretty well that this topic has been discussed very intensively (I read a lot all over the day).
Still, how probable is it that FB might allow me to create a frontend crawler for a non-commercial, non-public research university project?
My crawler should repeatedly lookup a very limited number of specific public fan pages and collect anonymized data like number of fans, status updates and their number of likes and number of comments each.
What I would like to show is what kind of topics in media pages are "liked" and discussed most and how that develops over time. I know about FB's restricted TOS. Thanks for your opinion on that.
The second question concerns technological approach / authorization: Reading a fan page's number of fans, status updates and their number of likes each - could I even use the API/OpenGraph for such a crawler? I think for reading page walls, you need an access token at any cost, and realizing an automatic "crawler" via an application therefore is not possible I guess (as apps only react to users' actions and cannot act like cron jobs for example)?
As you see, I am pretty new to FB development and logic. Thanks so much for your expertise.
If you mainly target public pages then you should be ok.
You need to have a facebook app and then you can authenticate as the app from your program.
You will get an app token with which you should be able to use in order to crawl public pages data.
If you check the documentation for the Page object you'll see in the tables (fields and connections) that most of what's in the Permissions column is either "No access token or user access_token" or "any valid access_token or user access_token", if you have the app token you're good.
Also, and I think this is something you'll be interested in, the Page object has the "talking_about_count" field.
So, yes you can do it, at least most of it.
As for the TOS, since all of this is perfectly ok and straight from their official documentation, there's no problem.
I want to create an iPhone app that displays (among other things) a specific Facebook wall. For a good user experience I didn't want an app that required the user to have a Facebook account and I didn't want to force the user to have to log in to Facebook to see the latest "news" in the app. I started out by getting the wall RSS feed and tried parsing it ... I can "see" all the data I need ... but that is getting complicated quickly and has too many variables that are making the final results less than stellar. I have read through the Facebook iOS programming tutorials and it seems to me like the SDK forces the user log in, which I don't like.
My question ... Is there a way to use the Facebook SDK with hard coded profile credentials to access a specific wall without forcing the user to login? If possible, is that a recommended approach? Any other ways to skin this cat?
I have read through the Facebook tutorial and searched through many postings on this site but haven't found an answer to this ... sorry if this a newbie question and has already been answered.
Item I.2. of the Facebook API policy list says
You must not include functionality that proxies, requests or collects
Facebook usernames or passwords.
It sounds to me like that's what you're proposing to do; i.e., the user will be able to see a certain wall, but using hard coded credentials (not their own). In other words, your credentials are proxying for the user.
I do not know if it is technically possible to do this (I imagine it is) but I don't think it's a good idea, and I do think it's a violation of the Facebook API terms of service.
First you need to get the a access_token by parsing your app id and secret.
https://graph.facebook.com/oauth/access_token?client_id=YOUR_APP_ID&client_secret=YOUR_APP_SECRET
Then send following request to get the data you want. Note that only public data will be accessible.
https://graph.facebook.com/FACEBOOK_USER_ID/?access_token=ACCESS_TOKEN
I'd like to know if there is a possibility to check (using Graph API or any other way) whether given user likes / shares a specific link. Probably I'll have this user's facebook ID or facebook login, but my site is non-Facebook application. Actually it's Dot Net Nuke portal (target: .NET with MS SQL Server) with part of it being avaliable as Facebook app, but certainly not greater part of it, so the solution should be out of Facebook Connect, although it's not a showstopper if it's necessary.
We'll be giving points to users who share/like most of links that we serve in our portal and such possibility would be a great help to make a ranking.
Another option we consider is making some kind of "wrapper" or proxy for FB like / share buttons which will at first save some data in our database (probably - this user clicked on like for this link) and then go on with standard FB like / share route. Did anybody of You tried such solution?
If You have any other suggestion on the subject, please, post them, we'll be really thankful.
It is possible to know if a user has LIKED a site or not. You can get all user's likes with Graph API (you need user_likes permission). Take a look at the docs: http://developers.facebook.com/docs/reference/api/user/
I'm not sure if you can know if he has shared your site, but you could try by parsing his wall with the read_stream permission and then look for your site name/URL post by post.
For just general liking of items on your site, you can use a Facebook Social Plugin. However, you won't be able to associate (or really even access) user activity with users on your site without integrating Facebook Connect and creating a Facebook application for your site. At that point you can design with greater control all the possible user activity and interleave with your facebook calls other calls that affect users' accounts on your site.
I'm a graduate student whose research is complex network. I am working on a project that involves analyzing connections between Facebook users. Is it possible to write a crawler for Facebook based on friendship information?
I looked around but couldn't find any things useful so far. It seems Facebook isn't fond of such activity. Can I rely on the Facebook API?
Update (Jan-08-2010): Thank you very much for the responses. I guess I probably need to contact Facebook directly then. Cheers
Update (Feb-16-2011): A new book, "Mining the social web", just came out. In it, there is a chapter devoted entirely for mining Facebook using Python. Cheers.
You can't rely on the Facebook API unfortunately. To get friend information, you need to use something like friends.get(). However, any Facebook API method that returns user information like this requires that you have an active session key from that user, and generally the way you get an active session key is to have the user come to your Facebook application or page.
In summary, the information you are talking about is essentially private. You can't pick a person from Facebook, get their friends, and get those friend's friends, and so on. To me this is a good thing for privacy, but of course it prevents arbitrary analysis.
I'd throw out the idea of writing a quick and dirty application with some user appeal that you could use for research. If a group like S**t My Dad Says (funny, not really safe for work) can get 120,000 users in a couple of months, you could probably plead your case with a small research application and get a reasonable amount of users.
The problem is that facebook friendship information is typically private and only accessible to friends. It should be a lot easier to build this network on Twitter, if this is an option for you.
As others have stated, this is typically private information. If, however, Facebook per se isn't a requirement, you could use Google's Social API. A snippet from the Google Social Graph API page: "With the Social Graph API, developers can now utilize public connections their users have already created in other web services. It makes information about public connections between people easily available and useful."
Here's an article on using it in Ruby:
http://articles.sitepoint.com/article/google-social-graph-api-ruby-rails#
This lifecode post provide a basic python script to scrape your facebook friends contact info.
The output of this script, is the profile ID, profile pame, profile URL, e-mail address and mobile/phone number (if provided by friend).
WARNING: This is against Facebook TOS. Use at your own risk.
Info provided for educational and research purposes
http://ruel.me/blog/2010/11/26/scrape-your-facebook-friends-contact-info-with-python/
You can use http://www.facebook.com/directory/ to get the public listed people.