Currently I have a dotnetcore WebApi that is serving up videos. The videos are stored in a SQL server table as a varbinary(MAX). This was working however I was reading that to support on IOS safari we needed to accept the ranges header, so I have added support for this (I think).
However now I am noticing two things (could be unrelated):
1) Whenever a call is made to this API the CPU throttles to 100%. I can only assume that is EntityFramework querying the db for a 25MB file. Seems crazy but the API is doing nothing else? Can this be improved as the server just grinds.
2) Multiple requests are made to the API with different range bytes requested. However my api in turn queries the db on each request and so sends the CPU into overdrive for a long period.
Is there a better way of handling range requests when querying for a large object?
If you ask me, EF is not really well suited for this, it's too clunky and resources consuming. You can write your own T-SQL using something like substring. This being said, from a practical point of view, depending on how many and how big these files are and how many users you have, I would not go with such a solution.
I don't think a SQL database should be how you store your data at all for this.
You could start doing some research on how netflix does it: https://www.techhive.com/article/2158040/how-netflix-streams-movies-to-your-tv.html
You probably want something like that, a CDN system, some sort of caching. Your way of doing it now might work while you build it, with one or two users but if this is an API used by lots of people, you will quickly find out that it won't scale.
So I'm in the middle of planning a little web app that will require quite large amounts of data stored on a user level, in one case, the system would take a large object from a system level and make a "user specific" version, a user can have multiple ones of these. Simplest would be to compare it to a form stored in a google spreadsheet, where the user is expected to use the template spreadsheet, then change not only the answers but also the question.
Security wise I am quite OK
In the second case there is requirement to store multiple objects, size about 250k to maybe 3mb, once again on a user specific level, with a potential to move it to a system level so additional users can access it. As an example, say the user can upload pictures, but may not want to share all of them. However, a user may choose to "publish" a small number of them because they are happy with those specific pictures.
What design patterns should I consider using specifically around web apps where the user have decent amounts of data? For example, would it make most sense to use a single large database and have a table that keeps track of resources or create separate tables per user?
I have considered putting it all in a mongo database.
Your approach may be wrong.
If you want to store user based binary data and make it accessible for the user itself or the community, you would need a hierarchic structure like so:
userid1
pic1,pic2,pic3
userid2
pic4,pic5,pic6
community
pic7,pic8
You could then grant read permissions to "community" for all users, and permission for each user to its own directory.
Usually there is nothing wrong using a database to store binary files if you consider partitioning, role permissions and an applicable interface to access the data.
My suggestion is to use a binary repository like Artifactory.
It provides hierarchic structures, simple search queries using HTTP requests and has caching abilities for frequently queried objects.
I also think that http requests are a lot easier to use and also there is an abstraction layer to the data which is more secure.
Artifactory is free.
There are several apps that I use on my Mac that store their data in core data. I can see the data I want in CoreDataPro. I want that data - specifically I want to send changes in that to some remote end points (such as Zapier, or some other REST service).
I was thinking of piggybacking something like RestKit - such that I provide a configuration file saying where the app is and what end points the data needs sending to. I need only to scrape the data and send to REST, not a two-way sync.
I'd prefer a utility that I could configure rather than having to code a Mac application.
I noted http://nshipster.com/core-data-libraries-and-utilities/ - RestKit still seemed the most capable, but in https://github.com/RestKit/RestKit/issues/1748 I was advised that coredata projects should only be opened by a single application at a time, and really RestKit is designed for baking into the source app (rather than for database scraping and sending).
What approach would you take?
I also noted:
http://www.raywenderlich.com/15916/how-to-synchronize-core-data-with-a-web-service-part-1
Thanks, Martin.
First, Core Data is an object store in memory. What is written to disk from Core Data can be in one of several formats. One of those formats happens to be SQLite. If the application is writing to SQLite then it is possible to sample that same file and push it somewhere else.
However, each application will have its own data structure so you would need to be flexible in the structure you are handling.
RestKit has no value in this situation as you are just translating objects into JSON and pushing them to a server. The built in frameworks do that just fine.
There is no utility to do this at this time. You would need to write it yourself or hire someone to write it.
If I were going to do something like this, I would write it using Core Data itself interrogate the model from the application that wrote the data in the first place and then translate the database into JSON and push it. You won't be able to tell what is new vs. old so the server will need to sort that out.
Another option, since you can't diff anyway, is to just push the sqlite file to the server and let the server parse through it.
Other answers might include:
use a middleware platform e.g. using rssbus.com (only) sqlite connections are free to send the events
as my target system (http://easy-insight.com) actually has a transmitter that sends new records it sees from MySQL abd PostgreSQL, I could https://dba.stackexchange.com/questions/2510/tools-to-migrate-from-sqlite-to-postgresql or use an ETL such as http://www.easyfrom.net (I did ask the vendor for SQLite support a long time ago, but SQLite is just not a priority for them).
I'm wondering whether a good answer (where good excludes Objective-C and includes languages that I do know, such as - to a limited extent - Ruby) is to use MacRuby and its Core Data libraries.
Core Data seemingly can be exposed as an Active Record. https://www.google.com/search?q=macruby+coredata , notably http://www.spacevatican.org/2012/1/26/seeding-coredata-databases-with-ruby/
However, MacRuby seems to have faded - https://github.com/MacRuby/MacRuby/issues/231 - it won't even compile on Mavericks.
Imagine we have 2 services: Product and Order. Based on my understanding of SOA, I know that each service can have its own data store (a separate database, or a group of tables in the same database). But no Service is allowed to touch the data store of another Service directly.
Now, imagine we have stored the product and order data independently inside Product and Order Services. In the Order Service, we can identify products by their ID.
My question is: With this architecture, how can I display the list of orders and product details on the "same" page?
My understanding is that I should get the list of OrderItems from OrderService. Each OrderItem has a ProductID. Now, if I make a separate call to ProductService to retrieve the details about each Product, that would be very inefficient.
How would you approach this problem?
Cheers,
Mosh
I did some research and found 2 different solutions for this.
1- Services can cache data of other Services locally. But this requires a pub/sub mechanism, so any changes in the source of the data should be published so the subscribing Services can update their local cache. This is costly to implement, but is the fastest solution because the Service has the required data locally. It also increases the availability of a Service by preventing it from being dependent to the data of other Services. In other words, if the other Service is not available, it can still do its job by its cache data.
2- Alternatively, a Service can query a "list" of objects from another Service by supplying a list of identifiers. This prevents a separate call to be made to the target service to get the details of a given object. This is easier to implement but performance-wise, is not as fast as solution 1. Also, in case the target Service is not available, the source Service cannot do its job.
Hope this helps others who have come across this issue.
Mosh
DB integration (which is really what you are talking about when two services share a table in a DB) is wrong at so many levels!
It completely breaks some of the major principles of software engineering
loose coupling,
encapsulation
separation of concerns
A service should be (to earn that name) completely independent, namely:
it must not rely on others to ensure the consistency and coherence of its data
it must not rely on others to guaranty the security of its data
it must not depend on external implementations (only interfaces)
Two services that share data at the DB level are unable to guarantee any of the former.
The fact that you "control" both services is completely irrelevant. Today you control... tomorrow you might want to outsource or replace one of the services. That should be as simple as ensuring the proper interfaces are in place.
Imagine both services that share a table with some field (varchar) in it. Now one service needs to change that field to numeric... bang the other service stops functioning - loose coupling goes down the drain.
Most of the time the trick lies in properly defining the service scope and clearly stipulating what a service do and what it doesn't do. You should also avoid turning everything into a service. Set your service granularity to high and services will start popping everywhere and integration headaches will escalate.
That being said, there are some situations where data integration between services poses some challenges. The main premiss do, should always be - data can belong only to one service. Data is intrinsically tied to business logic that affects data consistency and coherence and as such there should never be more than one service controlling any given data.
Another approach would be to have some sort of data source that lives outside of the SOA services. This data source could be considered your cache of the data, your operational data source or even a data warehouse. Extraction packages can export the data from the services (and/or some sort of real time mechanism). You can query this data source how you want.
The advantage of this approach is that the SOA black box is maintained and you can swap out a service knowing how you have coupled it.
Disadvantage is the added complexity and maintenance overhead.
SOA is just a buzz-phrase for deploying components behind web services. How many data stores you have is entirely up to you. In some cases it makes sense to have partitioned data behind individual components, and in other cases all the data lives behind one service, and in other cases many components that expose service interfaces connect to the same database via the database's connection protocol. Approach the problem by approaching the problem, not be imposing artificial constraints.
I don't think there is any principle in SOA that services should have separate data store. In general it is actually impractical. Yes you can have product and order service and the client can do the join using web service call as you said and this may be acceptable in some scenario. But that doesn't mean that you cannot have a specific service for a client if you already know the client's behaviour and performance requirements.
What I mean is that you should have a search service that returns orders and products with the join done in database. This is practical and would solve your business problem.
It is unfortunate to see this whole discussion being deteriorating in a "can I use a shared database or not in SOA" statement quest, which is totally irrelevant and does not help answering the original question at all.
More then often in a real world situation the data is already stored in different systems to start with. Customer data for example is coming from CRM, product data from SAP, contract data from yet another different source.
It is not a quest for getting this data technically together, rather than an understanding there is only one source of the data. To differently put it, there is only one owner of the data within your enterprise, who is solely responsible for maintaining it and ensuring the correct data quality.
Storing data locally for performance reasons means replicating data, which is more than often a dangerous venture, unless you have a solid caching strategy in place. I think Mosh has given some sensible answers when faced with an existing application landscape.
I'm planning my first iPhone app and I'd like to get some inputs as to how to build it, right from the start. The iPhone app is being built to be paired with a public facing web application that is already built in PHP.
I'd like the web platform to be central (data is housed in a mySQL database), and have the
iPhone clients talk to it and use REST'ful methods to perform the functions of the site
(fetching latest content, posting content, voting, account management as examples).
I'd like the clients to get a local copy of the data in a SQLite database, but refresh to get the latest version of the feed (similar to the Twitter app).
Couple of thoughts I have right now:
Use something like ASIHTTPRequest to send/recieve data to PHP files on the server listening for requests
JSON - would I be better off to send the GET/POSTS to a PHP that returns JSON objects, and work with some sort of wrapper that manages the data and communicates changes to the local SQLite database?
Am I totally off in how I should be building this thing to communicate with the web? Is
there a best practice for this?
I'd really appreciate any input on how you would architect this sort of a setup.
Thank you,
EDIT: After reading my own post again, I know it sounds like a Twitter client, but it is NOT, although it has similar features/structure of a Twitter type setup. Thanks!
As you already outlined in your plan, XML and REST are a great way to communicate with a web application. I want to suggest few details about how to actually design and build it, or what you should keep in mind.
First of all, I believe it's important to stick with MVC. I've seen people creating HTTP connections in view-controllers, controllers being NSXMLParser's delegate, controllers containing data in member variables. I've even seen UITableCells establishing HTTP connections. Don't do it!
Your model and its basic manipulation code should be as much extracted from user interface as possible. As you already have created the model in your web-application, try to recreate the entities in your iPhone project. Don't be afraid of having some simple methods in entity classes, but do not make them use external resources, especially tcp connections. As an example of methods in entity class you might have methods that formats data in specific ways (dates as an example, or returning fullname as concatenation of firstname and surname), or you can even have a method like - (void)update that would act as a wrapper to call class responsible to update the model.
Create another class for updating the model - fetching the XMLs from web-app. Do not even consider using synchronous connections, not even from a dedicated thread. Asynchronous connections with delegate is the way to go. Sometimes multiple requests need to be made to get all required data. You might want to create some kind of state-machine to keep the information about in which stage of downloading you are, and progress from stage to stage, skipping to the end if error occurs, re-executing from failed stage after some moments.
Download data somewhere temporarily, and first when you have it all, make a switch and update user interface. This helps responsiveness during launching the app - user gets to work immediately with data stored locally, while the update mechanism is downloading the new data.
If you need to download lots of files, try to download them simultaneously, if dependencies between files allow for that. This involves creating a connection per request, probably delegate instance for each of them. You can of course have only one delegate instance for all of those connections, but it gets a bit more complex to track the data. Downloading simultaneously might decrease latency considerably, making the mechanism much faster for the user.
To save the time and bandwidth, consider using HTTP's If-Modified-Since and/or ETag headers. Remember the time or tag when you requested the data the last time, and next time send it in HTTP's header. Your web-application should return HTTP code 304 if content has not been changed. iPhone app should react on this code accordingly in connection:didReceiveResponse:.
Create a dedicated class to parse the XML and update the model. You can use NSXMLParser, but if your files are not huge I strongly recommend TouchXML, it's such a pleasure to work with XML as document (it also supports XPath), instead of an event based API. You can use this parser also when files are downloaded to check their validity - re-download if parsing fails. That's when dedicated class for parsing comes handy.
If your dataset is not huge, if you do not need to persist downloaded data on iPhone forever, you probably don't need to store them in SQLite database, you can simply store them in XML format - just a simple caching. That at least might be the way for a twitter app. It gets easier that way, but for bigger data sets XML consumes lots of memory and processing power - in that case SQLite is better.
I'd suggest using Core Data, but you mention this is your first iPhone app, so I suggest you don't use it. Yet.
Do not forget about multitasking - your app can go to sleep in the middle of download, you need to cancel connections, and cleanup your update mechanisms. On app's wake-up you might want to resume the update.
Regarding the view part of the application - use Interface Builder. It might be painful in the beginning, but it pays off in the long run.
View controllers are the glue between model and views. Do not store data in there. Think twice about what to implement where, and who should call it.
This is not related to architecture of the app, but I want to remind that Objective-C is very expressive language. Code should read much like a sentence. Extend classes with protocols. As an example the other day I needed first line of a string. Sure, you can write a one-liner where you find first occurrence of a new-line, and get a substring from beginning till there. But it doesn't look right. I've added - (NSString*)firstLine into my NSString's protocol. Code looks so much better this way, it doesn't need any comments.
There are lots of things to consider in both architecture and design of any project, they both should go hand in hand. If one is causing trouble to the other, you need to adapt. Nothing is written in stone.
I'm currently working on an app that sounds similar to yours. I'd also suggest ASIHTTPRequest, and probably something like TouchJSON for JSON parsing, or extending/making a delegate of NSXMLParser if you want to parse XML.
As suggested by JosephH, depending on how your app works you may want to consider alternate authentication methods: I'd take a look at something token-based like OAuth, which has ready-made libraries for people to dig in to.
SQLite is totally viable for feed caching, although I prefer NSCoding so that you can freeze-dry your custom data structures.
As a general suggestion, make sure to spend a lot of time thinking about every use case and corner case for connections: it's easy to assume a user will only contact the server in certain ways and at certain times, and then after you throw in multitasking/incoming calls/lock screen/memory warnings, things can get hairy without any planning.
All in all, you seem to be on the right track, just make sure you plan out everything beforehand :)
Apple have a brand new in depth piece of sample code - MVCNetworking that shows in depth how to use subclasses of NSHTTPRequests and NSOperationQueues.
As others mentioned, I think you are asking the right questions and are heading in the right direction. All of the replies above are valuable advice. Here is my advice, and I hope you'll find it useful.
No matter which method/library you choose to talk to your web services, I think it's important to make a clean separation in the way you design your data model on the phone VS. the data model in your web application. You have 3 major distinctions to keep in mind for your design:
Data model on the web application (reflected by your existing mySQL database)
Since this is already there, there is not much to say about it, except that it will influence a lot your design for the following 2 parts. I suggest to make this model the 'master reference' for how your data is represented across platforms.
Data model on the iPhone app (reflected by the information you need to display in the iPhone app)
This is where the fun begins. First, you need a good understanding of what data you need to display in the phone app. So have a good, high level design of your app first (use pen and paper, draw mock-ups of each view and the interactions between them, model the navigation between your view controllers etc.). It really helps to understand the interactions between your view controllers and the various bits and pieces of data you want to show in the app. This will help you create the requirements for the data model on the phone. Based on these requirements, map the existing (web) data model to a new model, suited to your iPhone app. This new model may or may not include all tables and fields found in your web app. But the general representation of the 2 models should be very similar (e.g. relationships, data types, etc.)
Data model used to communicate between the 2 above (this is your 'data exchange protocol')
Once you have the 2 representations of your data above, you need to 'translate' from one to the other, both ways. Design your data exchange protocol to be as simple and compact as possible. You don't want to waste bytes on useless information, as transmissions over the network are costly. (As a side note, you might think of compressing the transmitted data later on, but it's just as important to have a good design from the beginning). It's probably best to begin with a protocol in which the metadata is the same as the one in your web application model (e.g. same relationships, names of tables, attributes, etc.). But remember, you'll only have to serialize/de-serialize those entities and relationships that you listed in point 2) above. So design accordingly. Your exchange protocol may also include session tokens, authentication info, a version number, or other metadata, if you need it.
Remember: your data exchange protocol is what will de-couple your web application and iPhone application models. I found that it's best to de-couple them because they may both evolve over time. The data model on the iPhone for example, may evolve a lot especially when you will find that you need to re-model some relationships or add/remove attributes from your entities in order to improve application responsiveness, or the user experience, the navigation, or whatever.
Since this is a whole concern in and by itself, well, you need to design a generic serialization/de-serialization mechanism on top of your (JSON/XML/whatever parser you choose) that is flexible enough to sustain the potential differences between your 2 data models. These differences might be: entity/attribute/relationship names, primary key identifier names, data types, attributes to ignore, and the list goes on. I would definitely implement a serializer/de-serializer utility class in the iPhone app, backed by a .plist configuration file containing all supported entities, concerns, aliases you might have. Of course, each model object should 'know' how to serialize, de-serialize itself and its relationships (i.e. the required object graph depth).
One last note, since you will end up with 2 representations of your data, you will need a way to uniquely identify an object on both sides. So for example, think of adding a uuid attribute to all data that needs to be exchanged, or use any other approach that suits your needs.
I am building an app that has similar requirements to yours, and these are the approaches I found to be best so far. Also, you might find this video useful (it inspired me a lot on how to implement some of the issues I mentioned above and is especially interesting if you're using CoreData) :
http://itunes.apple.com/ca/podcast/linkedin-important-life-lessons/id384233225?i=85092597
(see the lecture entitled "LinkedIn: Important Life Lessons on CoreData & GameKit (March 12, 2010)" )
Good luck!
It's quite a broad question, and I think you're going in the right way anyway, however I'll do my best to give some advice:
JSON, ASIHTTPRequest and POSTs to PHP scripts sound like a great way to go.
If the data is not really sensitive, I'd use http most of the time, and use https only for a login page that either sets a cookie or returns a "token" that you use in subsequent requests. (HTTPS can be quite slow over a 3G connection as the overhead in terms of number of packets to setup an SSL connection is higher than a plain TCP connection.)
You should make sure you correctly pass any data from the input to the PHP scripts to the database, to avoid any SQL injection attacks - ie. used parameterised SQL, don't create sql queries by doing "SELECT * from users where username="+$_GET['username']"
I would do this like I have done with a lot of AJAX web-page stuff. i.e.:
Have a URL on your server side package the information to be transmitted into XML format. (This can be through a CGI/PHP script or whatever). Your transmitting XML in the message body - so it's easy to human read and debug with a standard web browser.
Use the standard iPhone NSXMLParser methods to parse out the individual data fields from the XML doc, and write it back to your database. This method is equiped to both fetch the data from a URL and parse it in one call - like:
NSURL *xmlURL = [NSURL URLWithString:#"http://www.example.com/livefeed.cgi"];
NSXMLParser *myParser = [[NSXMLParser alloc] initWithContentsOfURL:xmlURL];
Walk through the data hierarchy with the NSXMLParser methods and populate your database accordingly.