How to view child nodes of a parent from panda dataframe? - mongodb

Question

I was trying to extract data from mongodb. So, I was using panda as a dataframe. I was using twitter dataset. The dataset was in json and when I import it in the database it looks like this:
user:Object
id:1292598776
id_str:1292598776
name:ahmd
screen_name:sameh7753
location:
url:null
description:null
protected:false
followers_count:5
friends_count:76
listed_count:0
created_at:Sat Mar 23 21:59:37 +0000 2013
favourites_count:1
utc_offset:null
time_zone:null
geo_enabled:true
lang:ar
contributors_enabled:false
is_translator:false
profile_background_color:C0DEED
profile_use_background_image:true
default_profile:true
default_profile_image:false
follow_request_sent:null
So, here 'user' is the parent and under it there are many children.There are other fields too in the dataset.
So, I was trying to execute a query which will find any tweet, tweeted on 2013 and the location of the tweet is "US". And then I was storing those cursors in the panda data frame. So when I was printing the data frame I was expecting to see those screen_name but it was not getting printed and also I couldn't access those data.
Here is the code I was using:
import pandas as pd
from pymongo import MongoClient
import matplotlib.pyplot as plt
import re
pd.set_option('display.expand_frame_repr', False)
def _connect_mongo(host, port, db):
conn = MongoClient(host, port)
return conn[db]
def read_mongo(db, collection, host, port):
""" Read from Mongo and Store into DataFrame """
# Connect to MongoDB
db = _connect_mongo(host=host, port=port, db=db)
cursor = db[collection].find({'created_at':{'$regex': '2013'}},
{'place.country':'US'}, no_cursor_timeout=True).toArray()
print cursor
# Expand the cursor and construct the DataFrame
df = pd.DataFrame(list(cursor))
return df
db = 'twittersmall' #'twitter'
collection='twitterdata' #'twitterCol' #
#query={'lang':'{$exists: true}'}
host='localhost'
port=27017
var = read_mongo(db, collection, host, port)
print var
It only prints under the user column in panda data frame this:
False {u'follow_request_sent':
u'profile_use_b...
And rest of the attributes don't get print and I can't even access them by writing var['user.screen_name'] stated in the python code.
How can I access the data?

Answer 1

First you have to include from pandas.io.json import json_normalize.
Now your read_mongo function should be like this-
def read_mongo(db, collection, host, port):
""" Read from Mongo and Store into DataFrame """
# Connect to MongoDB
db = _connect_mongo(host=host, port=port, db=db)
cursor = db[collection].find({'created_at':{'$regex': '2013'}},
no_cursor_timeout=True)
cursor = list(cursor)
df = json_normalize(cursor)
return df
Here json_normalaize flattens those fields which have children and make them columns of the panda dataframe.