I'm using Cloudant and I'm struggling to pull/replicate 600 documents from server to my iPhone. First, it's pretty slow because it has to go one-document-at-a-time, and Second Cloudant was giving me "timeouts" after the 100th-or-so REST request. (I have a ticket with Cloudant for this one, as it's unacceptable!)
I was wondering if anyone has found a way / hack to "bulk" replicate when pulling. I was thinking, perhaps it's possible to "zip up" all of the changes, send them in one file, and fast-forward the iPhone database to the last-change seq.
Any helps is great -- thanks!
Can you not hit _all_docs?include_docs=true to get everything in one shot? http://wiki.apache.org/couchdb/HTTP_Document_API#all_docs
I don't know couchcoccoa but it looks like the API supports this: http://couchbaselabs.github.com/CouchCocoa/docs/interfaceCouchDatabase.html#a49d0904f438587b988860891e8049885
Actually, why not make a view. Make a view that gives you your list and make sure your id is there. With your id, you can then go to the document and get all the rest of the required information that you need in order to update it if you need to.
There really is no reason you would ever need to hit every document individually. They have views and search2.0 for that. Keep in mind you are using a cloud based technology. This stuff is not sitting in your basement, you can't just hit it a million times per device in a few seconds and expect anyone to not notice and/or get upset (an exaggeration, yes I know).
What I do not understand is that you are trying to replicate it to an iPhone? Are you running apache and couchdb in your app? Why not just read the JSON data and throw it into a database. or just throw it into a file if it updates that much and keep overwriting it. There is so many options that are a whole lot less messy.
Related
I wanna ask you about lists and pagination in APIs
I want to build a long list in home screen that's mean this request will have a lot of traffic because it's the main screen and I want to build it in a good way to handle the traffic
After I searched about the way of how I gonna implement it
Can I depend on postgresql in pagination ? Or I need to use search engine like solr
If I depend on the database and users started to visit the app, then this request gonna submit a lot of queries on the database is this gonna kill the database ?
Also I'm using Redis to Cache Some data and this gonna handle some traffic but the problem with home screen the response it too large and I can't cache all of this response in one key in Redis
Can anyone explain to me what is the best way to implement this request for pagination .. the only thing I want is pagination I'm not looking to implement a full text search but to handle the traffic I read that search engine will handle it to not affect the database or kill it
Thanks a lot :D
You can do this seamlessly with the pagination technology we know in PostgreSQL. PostgreSQL has enough functions and capabilities to do this. (limit, offset, fetch)
But let me give you a recommendation.
There are several types of pagination.
The first type is that the count of pages must be known in advance. This technology is outdated and is not recommended. Because at this time you need to know the count of records in the table. But calculating count of records is a very slowing process, mainly in large tables.
The second type is that the number of pages is not known in advance. Information from the next page is brought in parts only if necessary. Just like Google, LinkedIn and other big companies use it. In this case, it is not necessary to calculate the count of any table.
We have a website at swalif.com which is like a news website based on forums. We are currently using a mysql database and things are getting slow. We decided to go with the Sphinx Search Server to speedup things and its been going quiet well.
Recently we heard of something called 'memcached' and just going through it we think we should look into it before moving to a search server completely.
My question is what are the pros and cons of using 'memcached' as it is a fairly new topic to us.
Thanking you
I just got my site set up with memcached a couple months back and it's amazing. The pros are rather obvious. It can be used to cache information that is, perhaps difficult to gather. The best example is an expensive mysql query. Check your slow query log, that would be a good starting point for things to parts to target. I had this one main page that took 2.5 seconds to echo from the server (horrible, I know). I had thought about changing the way it was written and it would have been very complicated. I put in memcached on the "difficult" parts of that page and now it's down to 0.001 seconds to parse. It's just amazing.
There is one main con that I've run into. If you update your content, you have to delete all associated keys related to that new content so that your front end will refetch the data and cache the new data. If not, you get stale content. I have tens of thousands of entries in my memcached and it's difficult to delete all the appropriate ones. If you don't, you'll get old content. One solution is to just set your key expiration duration to something short (24 hours). If you do that, you know that your site will reflect the newest content, at worst, 24 hours after a change. So if you can live with that, this problem is rather moot.
Bottom line, it's one of the best tools I've ever seen. It took me less than a day to install it on the lions share of my rather big site and the impact was tremendous.
I want to know what is the best way to store data on the iPhone from a web service.
I want the information to be stored on the device so the person doesn't need to access the web service every time he/she needs it. The currently information isn't much and contains less that 150 records. The records might update from time to time and a few new ones will be added. What is the best way to go about storing the data?
Thanks
If you use ASIHTTPRequest for your network stuff (and if you don't already, I can't sing its praises highly enough), you will find it has a cache layer built in which is perfect for situations like this.
You can activate it with a simple one line;
[ASIHTTPRequest setDefaultCache:[ASIDownloadCache sharedCache]];
And you have full control over the cache policy etc - just read the documentation.
The other simple approach of course is - on the assumption that your web service is returning JSON or XML - simply to store the response in a local file against a hash of the request parameters, then when you request the data again, you can first look to see if the file exists and if it does, return that data rather than going back to the website. You can roll your own cache policies etc too.
Since I discovered ASIHTTPRequest had a cache though, I've not needed to roll my own again.
I find that using coreData or sqllite3 is just overkill for 99% my requirements and a simple cache works very well.
If the data is relational, a Sqlite3 database would be the best storage option you have.
Also, this helps by allowing you to retrieve from the server and to update only the records that have changed, thus saving time and bandwidth.
This is the best option from a scalability point of view as well, as you stated that "current information isn't much", thus giving the impression that this is only a current situation, that may be subjected to further change, probably towards more records being added in time.
Sqite3 also gives you more control and better performance than using, for instance, Core Data. Here's an article explaining some of the details. Moreover, if you work through an Objective-C wrapper, such as FMDB, you get all the advantages without managing the complexity yourself.
Say I have a TODO list iphone app, that can be edited/viewed from both a web application and the iphone application.
When on the iphone, when a user views all his todo lists, or sub-items, I would think that each time the user views a particular list it shouldn't be hitting the web applications API every-time, but rather cache locally the values and only hit the web when things change.
What strategies are there for this type of scenerio?
I agree with you in your dirty-otherwise-do-not-contact-the-server point. And I think this point is pretty straightforward and easy to implement.
However, be careful in this scenario: it gets dirty but at the same time, the device cannot reach the internet. In this scenario, I suggest you check the internet accessibility on a frequent basis (even when your app is in the background), and try to reach your server and update whenever possible.
This is a tricky problem. I'm currently working on an app that needs to perform a similar synchronization, and I haven't decided how I want to handle it yet.
You're right in that you don't want to be hitting the web repeatedly. It would slow the app down considerably. Keeping a local cache is the way to go.
One drawback is that the user could change/add an item on the web and you wouldn't see it on the phone. You'd need to have a refresh button (like in the Mail application, for example) to allow the user to get the changes.
Then you have an issue with conflict resolution. Say the same item is edited on both the phone and on the web. How does the user pick which one to keep, or do they get duplicated?
I think the best way to do this is to replicated your server's schema in CoreData. Then load a given element from the local DB, and in the background go out and check that element for updates if the device has an internet connection. You're hitting the db each time, but the user is not slowed down by the process.
You should not query the internet everytime you view the list.
But when you make updates to it, or edit it, you should update the server as well. That will make your life a whole lot simpler. That way when the user updates an item that he deleted in the web server, the server will just throw that request out...
I am writting a django app and Iphone app, I need to keep them in sync.
Users can delete, update and create new objects in the web app, and in the iphone app.
When they get online with the iphone both app must be in sync.
Is there simple way to do this?
Thanks,
Joaquin
In general: There's no simple way. But I'll outline an approach.
If you don't care about changes being overwritten: Keep a timestamp of the most recent change to each record, and a timestamp of each sync. When syncing, you get a list of all updates on the iPhone since the last sync, and all updates on the server. You write from the iPhone to the server if the iPhone timestamp for that record is newer than the server one, and vice versa.
But you probably care. Say you've edited a note called "Where to meet up on Friday." It started out empty. Now, on the phone, you've written, "My house." Ten minutes later, your friend edits the same note on the server and writes, "The diner." Who wins out? Stack Overflow can't answer that for you; it's application-specific.
OK, so modify the approach above: if both the server version of a record and the local version have been edited since the last sync, then you have to ask the user what to do. That's the basic algorithm.
If you care a lot about changes not being overwritten, to the point that you want to merge changes to different places in the same documents, then your system will begin to approach the complexity of version control systems like Subversion or Git. Not at all simple.
There's no built in way to do this. You need to keep a server data store, and a local data store on the iPhone, and when online, check the differences manually, and see what action you should take on the server and the iPhone side (delete, update, etc.).
Sync is usually hard. I suggest you start laying out the server and iPhone data stores, and think how they relate, and how can the server or the iPhone know the status of their counterpart record, so to keep them in sync.