I'm a graduate student whose research is complex network. I am working on a project that involves analyzing connections between Facebook users. Is it possible to write a crawler for Facebook based on friendship information?
I looked around but couldn't find any things useful so far. It seems Facebook isn't fond of such activity. Can I rely on the Facebook API?
Update (Jan-08-2010): Thank you very much for the responses. I guess I probably need to contact Facebook directly then. Cheers
Update (Feb-16-2011): A new book, "Mining the social web", just came out. In it, there is a chapter devoted entirely for mining Facebook using Python. Cheers.
You can't rely on the Facebook API unfortunately. To get friend information, you need to use something like friends.get(). However, any Facebook API method that returns user information like this requires that you have an active session key from that user, and generally the way you get an active session key is to have the user come to your Facebook application or page.
In summary, the information you are talking about is essentially private. You can't pick a person from Facebook, get their friends, and get those friend's friends, and so on. To me this is a good thing for privacy, but of course it prevents arbitrary analysis.
I'd throw out the idea of writing a quick and dirty application with some user appeal that you could use for research. If a group like S**t My Dad Says (funny, not really safe for work) can get 120,000 users in a couple of months, you could probably plead your case with a small research application and get a reasonable amount of users.
The problem is that facebook friendship information is typically private and only accessible to friends. It should be a lot easier to build this network on Twitter, if this is an option for you.
As others have stated, this is typically private information. If, however, Facebook per se isn't a requirement, you could use Google's Social API. A snippet from the Google Social Graph API page: "With the Social Graph API, developers can now utilize public connections their users have already created in other web services. It makes information about public connections between people easily available and useful."
Here's an article on using it in Ruby:
http://articles.sitepoint.com/article/google-social-graph-api-ruby-rails#
This lifecode post provide a basic python script to scrape your facebook friends contact info.
The output of this script, is the profile ID, profile pame, profile URL, e-mail address and mobile/phone number (if provided by friend).
WARNING: This is against Facebook TOS. Use at your own risk.
Info provided for educational and research purposes
http://ruel.me/blog/2010/11/26/scrape-your-facebook-friends-contact-info-with-python/
You can use http://www.facebook.com/directory/ to get the public listed people.
Related
I know that you want to suggest me private api or any scraper. I used them but they don`t return accounts geolocation. In my application I want to get instagram accounts geolocation, likes, comments, followers and follows in order to get engagement of account.
If you say it is not possible look TrendHero website it returns all information about users, followers rank, country information and other handy information which you can find in your insights (in instagram account) without users permissions.
My first impression, without caring to dive deep into the company you linked, is they are simply using scrapers. That's typically how those big contact databases are done.
The Instagram APIs simply don't provide the access they used to. That's how it is. Simply asserting there must be a way doesn't mean there actually is.
#WizKid Instagram can't actually prevent scraping. They just make a little fuss about it.
I need to initiate searches on facebook marketplace from my application on the user's device. This needs to happen on the user's device, and as the facebook user associated with the user using the application, to avoid getting blocked by facebook. As far as I can understand, this cannot be achieved using facebook's OAuth login and accessing the facebook information that is accessible through it.
Another hypothetical way that comes to mind, is to use the token that the user uses to login into facebook itself, though this one sounds frankly illegal.
tl;dr is there a way to run a search and retrieve the results on facebook marketplace as a user on a user's device?
I found your question on SO, and spent a good bit of time researching this, as I wanted to know the answer myself. Unfortunately, as best as I can tell, this is not possible through any official APIs. It seems like they may have once had something like this, as I found several links to FB API documentation that looked promising. But, the links all rerouted back to the Graph API homepage. I found a couple of references to the big data breaches FB has suffered in the last couple of years as possible reasons why their APIs were retooled and locked down.
I found several Facebook developer posts with questions about a Marketplace API where there were either no answers after months or years, or an official Facebook moderator posting a response like "This is a great idea, we always want to improve, use this form to submit your idea" and so on with no follow up.
I also found at least one SO post within the last 18 months where someone in the comments claimed to be able to post products to FB, but I think this was related to the Business Page Product Catalog, and is not what you're looking for. This is more like if a car dealership or something wants to post a new car for sale that's tied to their FB business page.
The Graph API allows for some decent searching and edge traversal but it is all related to posts, pictures, feeds, etc., and nothing related to the marketplace. Facebook pushes their Marketing APIs so heavily that it was tough to filter through that noise. And, of course, all of marketing apis are geared toward creating ads.
I found some Facebook information around API URLs looking like https://graph.facebook.com/search=terms&type=some_type that was very promising. But the type options seem to be limited to adcountry, adeducationschool, adeducationmajor, adlocale, adworkemployer, adkeyword, adzipcode, adgeolocation, and audienceinterest. And, as I dug deeper, it appears this is related to finding targeting groups for creating targeted posts. Nothing for marketplace.
I think that the answer, unfortunately, is that there are no offical FB APIs at this time that will allow querying search results from the Facebook Marketplace, much less provide enough information to reproduce a listing to display on a 3rd party app.
I'm a graduate student whose research is complex network. I am working on a project that involves analyzing connections between Facebook users. Is it possible to write a crawler for Facebook based on user's post information?
It is an unusual question, but..
I'm looking for someone who can do it together (if necessary even paid).
Building a Facebook crawler is against the TOS of Facebook:
https://www.facebook.com/legal/terms (3.2)
You will not collect users' content or information, or otherwise access Facebook, using automated means (such as harvesting bots, robots, spiders, or scrapers) without our prior permission.
I know pretty well that this topic has been discussed very intensively (I read a lot all over the day).
Still, how probable is it that FB might allow me to create a frontend crawler for a non-commercial, non-public research university project?
My crawler should repeatedly lookup a very limited number of specific public fan pages and collect anonymized data like number of fans, status updates and their number of likes and number of comments each.
What I would like to show is what kind of topics in media pages are "liked" and discussed most and how that develops over time. I know about FB's restricted TOS. Thanks for your opinion on that.
The second question concerns technological approach / authorization: Reading a fan page's number of fans, status updates and their number of likes each - could I even use the API/OpenGraph for such a crawler? I think for reading page walls, you need an access token at any cost, and realizing an automatic "crawler" via an application therefore is not possible I guess (as apps only react to users' actions and cannot act like cron jobs for example)?
As you see, I am pretty new to FB development and logic. Thanks so much for your expertise.
If you mainly target public pages then you should be ok.
You need to have a facebook app and then you can authenticate as the app from your program.
You will get an app token with which you should be able to use in order to crawl public pages data.
If you check the documentation for the Page object you'll see in the tables (fields and connections) that most of what's in the Permissions column is either "No access token or user access_token" or "any valid access_token or user access_token", if you have the app token you're good.
Also, and I think this is something you'll be interested in, the Page object has the "talking_about_count" field.
So, yes you can do it, at least most of it.
As for the TOS, since all of this is perfectly ok and straight from their official documentation, there's no problem.
I want to create an iPhone app that displays (among other things) a specific Facebook wall. For a good user experience I didn't want an app that required the user to have a Facebook account and I didn't want to force the user to have to log in to Facebook to see the latest "news" in the app. I started out by getting the wall RSS feed and tried parsing it ... I can "see" all the data I need ... but that is getting complicated quickly and has too many variables that are making the final results less than stellar. I have read through the Facebook iOS programming tutorials and it seems to me like the SDK forces the user log in, which I don't like.
My question ... Is there a way to use the Facebook SDK with hard coded profile credentials to access a specific wall without forcing the user to login? If possible, is that a recommended approach? Any other ways to skin this cat?
I have read through the Facebook tutorial and searched through many postings on this site but haven't found an answer to this ... sorry if this a newbie question and has already been answered.
Item I.2. of the Facebook API policy list says
You must not include functionality that proxies, requests or collects
Facebook usernames or passwords.
It sounds to me like that's what you're proposing to do; i.e., the user will be able to see a certain wall, but using hard coded credentials (not their own). In other words, your credentials are proxying for the user.
I do not know if it is technically possible to do this (I imagine it is) but I don't think it's a good idea, and I do think it's a violation of the Facebook API terms of service.
First you need to get the a access_token by parsing your app id and secret.
https://graph.facebook.com/oauth/access_token?client_id=YOUR_APP_ID&client_secret=YOUR_APP_SECRET
Then send following request to get the data you want. Note that only public data will be accessible.
https://graph.facebook.com/FACEBOOK_USER_ID/?access_token=ACCESS_TOKEN