I pull data (a list of properties) from a city's website and it contains info like address, property owner, and longitude and latitude.
I need to make a unique id out of this and the properties will be stored in a database. The id needs to be reproducible because I will be checking occasionally to see if there are new properties so the only way to cross compare is by being able to make the id on the spot.
Is this a common problem to run into when pulling data and checking if there is anything new? I am using mongoose as the database and I am not sure if a non relational database is the best solution for my problem but I know I can make it work.
The current plan is to just combine longitude and latitude and call that the unique id. The city's data unfortunately doesn't have any unchanging unique id so I have to hack something together for comparing their data to what I have saved in the database.
Any ideas on what a better less hacky method is? Does mongoose have a different way to deal with this?
I have a collection in mongodb that stores activities of customers like product_view, added_to_cart etc with productId. I need this data to display products to my customer when he visits next.
Right now I am thinking to store all data of a customer in a single document,such as with customer_id as key and corresponding activities in array like product_view activities in product_view array etc.This will be fast to fetch for me as all data of a customer will be in one key only, but my consideration is that data size will go on increasing always this way. Moreover I may need to check say last 50-100 activities of a customer only. For that too I need to fetch the entire document.
What will be the best way to store this data. Request for data will be very very frequent. How can I manage response time ?
Your question answers itself. Have customer activity as separate collection with reference to customerId. Any time customer visits, you know customerId, hence can apply filter/aggregate operations to get whatever you want.
This way you can do paginated fetch of customer activities.
I don't understand one thing about Cassandra. Say, I have similar website to Facebook, where people can share, like, comment, upload images and so on.
Now, let's say, I want to get all of the things my friends did:
Username1 liked you comment
username 2 updated his profile picture
And so on.
So after a lot of reading, I guess I would need to do is create new Column Family for each single thing, for example: user_likes user_comments, user_shares. Basically, anything you can think off, and even after I do that, I would still need to create secondary indexes for most of the columns just so I could search for data? And even so how would I know which users are my friends? Would I need to first get all of my friends id's and then search through all of those Column Families for each user id?
EDIT
Ok so i did some more reading and now i understand things a little bit better, but i still can't really figure out how to structure my tables, so i will set a bounty and i want to get a clear example of how my tables should look like if i want to store and retrieve data in this kind of order:
All
Likes
Comments
Favourites
Downloads
Shares
Messages
So let's say i want to retrieve ten last uploaded files of all my friends or the people i follow, this is how it would look like:
John uploaded song AC/DC - Back in Black 10 mins ago
And every thing like comments and shares would be similar to that...
Now probably the biggest challenge would be to retrieve 10 last things of all categories together, so the list would be a mix of all the things...
Now i don't need an answer with a fully detailed tables, i just need some really clear example of how would i structure and retrieve data like i would do in mysql with joins
With sql, you structure your tables to normalize your data, and use indexes and joins to query. With cassandra, you can't do that, so you structure your tables to serve your queries, which requires denormalization.
You want to query items which your friends uploaded, one way to do this is t have a single table per user, and write to this table whenever a friend of that user uploads something.
friendUploads { #columm family
userid { #column
timestamp-upload-id : null #key : no value
}
}
as an example,
friendUploads {
userA {
12313-upload5 : null
12512-upload6 : null
13512-upload8 : null
}
}
friendUploads {
userB {
11313-upload3 : null
12512-upload6 : null
}
}
Note that upload 6 is duplicated to two different columns, as whoever did upload6 is a friend of both User A and user B.
Now to query the friends upload display of a friend, do a getSlice with a limit of 10 on the userid column. This will return you the first 10 items, sorted by key.
To put newest items first, use a reverse comparator that sorts larger timestamps before smaller timestamps.
The drawback to this code is that when User A uploads a song, you have to do N writes to update the friendUploads columns, where N is the number of people who are friends of user A.
For the value associated with each timestamp-upload-id key, you can store enough information to display the results (probably in a json blob), or you can store nothing, and fetch the upload information using the uploadid.
To avoid duplicating writes, you can use a structure like,
userUploads { #columm family
userid { #column
timestamp-upload-id : null #key : no value
}
}
This stores the uploads for a particular user. Now when want to display the uploads of User B's friends, you have to do N queries, one for each friend of User B, and merge the result in your application. This is slower to query, but faster to write.
Most likely, if users can have thousands of friends, you would use the first scheme, and do more writes rather than more queries, as you can do the writes in the background after the user uploads, but the queries have to happen while the user is waiting.
As an example of denormalization, look at how many writes twitter rainbird does when a single click occurs. Each write is used to support a single query.
In some regards, you "can" treat noSQL as a relational store. In others, you can denormalize to make things faster. For instance, PlayOrm's #OneToMany stored the many like so
user1 -> friend.user23, friend.user25, friend.user56, friend.user87
This is the wide row approach so when you find your user, you have all the foreign keys to his friends. Each row can be different lengths. You may also have a reverse reference stored as well so the user might have references to the people that marked him as a friend but he did not mark them back(let's call it buddy) so you might have
user1 -> friend.user23, friend.user25, buddy.user29, buddy.user37
Notice that if designed right, you may NOT need to "search" for the data. That said, with PlayOrm, you can still do Scalable SQL and do joins(you just have to figure out how to partition your tables so it can scale to trillions of rows).
A row can have millions of columns in it or it could have just 10. We are actually in the process of updating alot of the documentation in PlayOrm and the noSQL patterns this month so if you keep an eye on that, you can also learn more about general noSQL there as well.
Dean
Think of each DB query as of request to the service running on another machine. Your goal is to minimize number of these requests (because each request requires network roundtrip).
Here comes the main difference from RDBMS paradigm: In SQL you would typically use joins and secondary indexes. In cassandra joins aren't possible, since related data would reside on different servers. Things like materialized views are used in cassandra for the same purpose (to fetch all related data with single query).
I'd recommend to read this article:
http://maxgrinev.com/2010/07/12/do-you-really-need-sql-to-do-it-all-in-cassandra/
And to look into twissandra sample project https://github.com/twissandra/twissandra
This is nice collection of optimization technics for the kind of projects you described.
I am making Map application and I have already able to call web service which show Shops location on map with annotation. My web service contain more than 200 shops in Australia. I am taking one UISearch bar in which when I insert Syd.....then Sydney, Australia kind of addresses autofill tableview should open. How can I achieve this.. Do I need to call web service Url again for address autofill tablview or Should I insert manually address or is there any method???
Edited:=
In my application I am searching shops location within 5,10,15Km of radius. And if user want another location instead of current location then he can insert his any location to find shops information around that inserted location. So when user insert his location then autofill address tableview should open
One possible solution can be to keep the predefined values in database (Core Data) and then on textDidChange delegate of searchBar you can query the database to search for a list if data matching the current serach keyword. The results thus produced can be shown in a table view with some animation effect so as to give the feel of auto suggest.
Calling the webservice on textDidChange will block the User Interface and will not be good option to proceed with. though this feature is more prevalent on web but on device I find the first choice more productive than the second one.
EDIT: Answer for problem being asked in edit part of question
You need to put an auto suggest.
Fetch data from Server (using web service).
Now to provide functionality of address suggestion like "Syd" turning into Sydney, Australia , my above answer will help you to put these static addresses in database and then providing user with auto suggest options. For the second part you can save the lat/long of the places in database and once user finalize his selection you can query the webservice to get you the data.
The steps can be summarized as ..
User Types Syd.
You query database to search for place which match Syd using some query such as place LIKE %syd%.
Populate the table with autosuggest with showing place name like
Sydney,Australia.
User selects a place , correspondingly you pick the Lat/long (fetched
along with the names in above query)and query your web service to
fetch you data for
Place = Sydny, Australia (Not really Required)
Latitude = SOME VALE for Sydney
Longitude - SOME VALUE for Sydney
Radius = 5,10,15 depending upon your application logic
Server will then fetch you all shops in given range for particular Lat/long.
More calculation intensive work can be done at server's end and at client's end less web service calls should be made to avoid latency.
I was having same app which was showing addresses of persons so I was using a webservice which was giving me all the addresses and I was keeping them in NSMutableArray and then I was checking the string in array in textDidChange function of UISearchBar and if there is any string matching the prefix then I was showing the address in UISearchBar...But in this case if you wrote only 'S' then it will return more than one record so there you need to decide according to client's requirement how to show, does he wants list or any matching record...If list is expected then you need to show the tableview and when user picks the value then just hide the tableview.....
I want to access a webservice to fetch alot of data (e.g. product lists/details/search results) and display this.
Are there any best practices for this type of operations?
Performance-wise, is there any better way than retrieving, parsing and displaying text data on each request and maybe load images in the background? Are there any wise caching policies which may be applied?
If I were doing something like this from the ground-up, here's what I'd do:
Have the web site post all the data in XML. Except for maybe the pictures - just have an XML field specify a URL for each picture. So, for example, say I was doing a product list.
Use NSXMLParser to both fetch and parse the XML data.
Use a separate NSData dataWithContentsOfURL: call to fetch the contents of each image, with the URL from the XML data
Write the XML data, (and the NSData image) to a database table with CoreData. Add a indexed timestamp field to the table.
You can now use the timestamp field to keep the newest "x" records in the database - and can purge the older ones if/when you need to.
Use the contents of the database table to populate a UITableView - or however else you want to present.
Make some sort of "next", "prev" or "update" fields in the UITableView to get more data from the web, if you need to display more data than is what is cached - or you want to update the data in the cache.