I've recently decided to take on a pretty big software engineering project that will involve developing a client-server based application. My plan is to develop as many clients as possible: including native iPhone, Android and Blackberry Apps as well as a web-based app.
For my server I'm planning on using a VPS (possibly from slicehost.com) running a flavor of Linux with a MySQL database. My first question is what should be my strategy for clients to interface with the server. My ideas are:
HTTP-POST or GET based communication with a PHP script.
This is something I'm very familiar with - passing information to a PHP script from a form, working with it and returning output. I'm assuming I'd want to return output to clients as some sort of XML or JSON based string. I'm also assuming I'd want to create a well defined API for clients that want to interface with my server.
Socket based communication with either a PHP script, Java program, or C++ program
This I'm less familiar with. I've worked with basic tutorials on creating a script or simple application that creates a socket, listens for a connection and returns data. I'm assuming there is far less communication data-overhead with this method than an HTTP based method. My dream is for there to be A LOT of concurrent clients in use, all working with the server/database. I'm not sure if a simple HTTP/PHP script based communication design can scale effectively to meet the needs of many clients. Also, I may eventually want the capability of a Server-Push to clients triggered by various server events. I'm also unsure of what programming language is best suited for this. If efficiency is a big concern I'd imagine a PHP script might not be efficient enough?
Is there a commonly accepted way of doing this? For me this is an attempt to bridge a gap between some of my current skills. I have a lot of experience with PHP and interfacing with a MySQl database to serve dynamic web pages. I also have a lot of experience developing native iPhone applications (however none that have had any significant server-based communication). Also I've worked with Java/C++, and I've developed applications in both languages that have interfaced with MySQL.
I don't anticipate my clients sending/receiving a tremendous amount of data to/from a server. Something on par with a set of strings per a given client-side event.
Another question: Using a VPS - good idea? I obviously don't want to pay for a full-dedicated server (slicehost offers a VPS starting at ~ $20/month), and I'm assuming a VPS will be capable of meeting the requirements of a few initial clients. As more and more users begin to interface with my server, I'm assuming it will be possible to migrate to larger and larger 'slices' and possibly eventually moving to a full-dedicated server if necessary.
Thanks for the advice! :)
I'd say go with the simplicity of HTTP, at least until your needs outgrow its capabilities. (The more stateful your application needs to be, the less HTTP fits).
For low cost and scalability, you probably can't go wrong with a cloud like Rackspace's or Amazon's. But I'm just getting started with those, my servers have been VPSs from tektonic until now.
Related
I'm wondering if for simply communicating with a PLC, like reading and writing tags, do I need all of the other heavy lifting that comes with an OPC-UA server?
I've tried writing a simple server in Python that talks to the PLC, but I get denied when requesting information from the PLC.
The Controllogix PLC I'm attempting to communicate with uses Ethernet/IP to communicate, so why doesn't a simple server/client script work? What is required exactly to communicate with an Allen Bradley PLC or PLC's in general?
There is quite a bit required to communicate with a PLC.
Each vendor has a driver, there are firmware compatibility considerations. Different protocols to think about.
OPC-UA makes it a bit more generic, but OPC-UA still has a set of things to work around when setting up communications.
Most of the OPC products I've worked with, needs to have their security adjusted to allow anonymous communication. It's generally bad practice to do this. (A network intrusion would be able to read/write to your automation layer) There is certificate signing and some encryption business that needs to be turned off if you're looking for simple communication. (Again, not a good practice but ok for learning)
After all that you have to have a notion of how your PLC is set up on your OPC server, there are channels, devices, namespaces etc. You'll point the OPC client to some opc.tcp://:
If you got this far you're almost done, I'm assuming your OPC server is running and has tags configured at this point. You can use your OPC-UA API to do a read. It can return just the value, or you can get an object back with tag health, timestamp, and a bunch of other data. Depends on the implementation. After that you can do subscriptions, writes...whatever else you need.
TLDR: OPC server not required, but may be the easiest method. Turn off security. (But turn it back on before exposing your control layer to the net)
I am also a little late to this conversation. If you are interested in coding your own solutions and don't want to use any of the commercially available standards, AdvancedHMI is a "mostly" open source solution written in VB.NET which is 100% free and provides communications to many different PLCs including the ControlLogix platform. Since I see you are programming in Python you may also be interested to know that the project does work under Mono on the Linux OS. I have used it to write gateways between EthernetIP and ModbusTCP and to pull data serially from OEM devices and push this data to a CLX PLC.
The forum is full of many helpful hints and is very active and supported.
Just trying to give you another option. DDE, NetDDE, FastDDE, OPC, DCOM, Suitelink.... These are all good, but mostly a pay to play adventure. As a programmer, it seems ridiculous to have to pay such an excessive amount of money just to talk to my hardware, IMHO. Sorry for the rant. Have Fun!
Update - Just wanted to also suggest the following open source project written in python:
https://github.com/dmroeder/pylogix
I have used this to write small programs for communicating with CompactLogix and ControlLogix. (Even to/from a RaspberryPi!)
Depends on several factors, if you want something simple to program you can opt for Modbus/TCP I think some AB PLC supports it without extra hardware.
However if you want something with more security for example for industrial use then OPC UA would be better choice but the programming has a complexity far higher than Modbus, even using the libraries of OPC Foundation or others. There is the option of using a commercial or free (if any) OPC UA server to save work, then you will need to program the client side only.
With Ethernet/IP it should also be possible, but the problem is that there is no clear specification and even different AB models talk different Ethernet/IP dialect ! , it is also far more complex to program than Modbus.
I am a little late to this discussion, but there are a couple commercial tools that make this a bit easier. The one that comes to mind for me when you say you are using python is Cogent's data hub. It is certainly not the cheapest tool out there, but they have already done all of the heavy lifting for PLC/PC communications & security.
If trying to read CLX data using Python, there are several open source implementations that will save you a lot of work. Such as this:
https://github.com/dmroeder/pylogix
If you use .NET and Visual Studio, you can use AdvancedHMI
to be able to read and write OPC Tags to ControlLogix platform is done via its communication Driver RSLinx. RSLinx acts as an OPC Server, it will need to be configured to communicate with the PLC and run on a networked PC on the same LAN. Several flavours of RSLinx are available (for WAN/VLAN also) but essentially this is the communications driver you need to talk to AB PLC's
There are must be a lot of apps that are designed to communicate with server. My question is only about App installed on iOS device + Server side service interaction. Web app is not what I am talking about, and there should be no webpage involved in this discussion at all. Typical examples are Apps like Instagram and Twitter, in which most of the information exchanged between the App and the server is just data like String, Image and Integers(wrapped in JSON or XML), no webpage presentation needed.
My question will be: if you are an independent app developer, and you are designing such an app from scratch without any existing website API, database structure or application(so you are not limited by any existing API or database structure or application protocol), What will be the most efficient approach?
What the sever side need to do are:
receive data send by the App;
process the data with designed logic;
interact with database(like MySQL);
do necessary data mining and analysis---this could be a constantly running service or one time task requested by the App client;
send the data back to the App upon request or spontaneously;
exchange or broadcast the data between/among different App clients (i.e.: group chatroom and peer to peer message);
As far I as know there are 3 obvious options to implement the server side:
PHP
Python
Ruby on Rails
(please feel free to add more options)
My questions are:
which one is the most appropriate choice to implement the server side?
If the App is focusing intensively on natural human language/text searching, analyzing and data mining, which one is the best choice? I heard Python is doing pretty good in this area.
Any advice on the database choices? I am using MySQL for now, and I found it's quite powerful for my purposes, I heard Twitter is switching to Cassandra. Will that be too difficult to start with?
For the server end, if you need to build a Server management interface, for you as an admin to manage and monitor the community, membership, data and such, is there any existing solution, or framework or tool for that? what will be the most efficient approach?
If a new programmer has no experience in non of them, which one you suggest he/she to start with?
Is there any good reference material or sample code on the server side in such context we can learn from?
I know there are a lot of very experienced experts on these areas on Stackoverflow, but I saw more newbies who just entered the iOS developing area without much knowledge in server/database programming experience. And I hope this thread can help these who are thinking to design an App with server/client structure but have no idea where to start with.
ps: I will keep updating this question thread and adding my findings on this topic, to help all other users at stackoverflow. :-) Please try to make your answer informative, easy to understand, and constructive. I guess most of readers for this thread will be new members of this great community.
Are you sure you want to spend time & money to develop your own Server & develop your own API?
There are lots of mBaaS (mobile Backend as a Service) providers today such QuickBlox, Parse,StackMob, which are ready to use and they have great Custom Objects API and some of predefined modules. They have great free plans with big quota. Some of them such QuickBlox has Enterprise plan - so you can buy license and they server team update server for you purpose.
So, i recommend not develop your server and think about mBaaS market.
Just about your issue - I can recommend look at QuickBlox Custom Objects code sample and also Custom Objects API. Custom Objects module provides flexibility to define any data structure(schema) you need. Schema is defined in Administration Panel. The schema is called Class and contains field names and their type. I think it's what you need.
which one is the most appropriate choice to implement the server
side?
Well that depends on what you know, there is reason to choice one of the other
If the App is focusing intensively on natural human language/text
searching, analyzing and data mining, which one is the best choice? I
heard Python is doing pretty good in this area.
This would reflect on your first question, you pick the language on you needs. Thus if python makes it easier then pick that one.
Any advice on the database choices? I am using MySQL for now, and I
found it's quite powerful for my purposes, I heard Twitter is
switching to Cassandra. Will that be too difficult to start with?
Again not one that is easy to answer, since it all has to do with requirements. But any SQL server will do. Cassandra is meant for "scalability and high availability without compromising performance" accourding to there website. Do you think you webservice will get many request then it might be a choice to consider.
For the server end, if you need to build a Server management
interface, for you as an admin to manage and monitor the community,
membership, data and such, is there any existing solution, or
framework or tool for that? what will be the most efficient approach?
This again is only going to be answered when you pick the a SQL server and server language.
If a new programmer has no experience in non of them, which one you
suggest he/she to start with?
Start with something simpler, you are really going out on a limb here.
Is there any good reference material or sample code on the server
side in such context we can learn from?
Propably there is some, but you should really start small and work from there.
Twitter started out as a Rub on Rails app and is working on scalability and availability which ruby is not really good ar (that is my person opinion). or Look at facebook they have written a php to c compiler to make php run faster.
The only thing I can say to start code, when you app does take off then tackle the some of the performance issues.
And since you state that you are new to programming do not bite of more then you can chew.
This is a huge question and I don't think there is a best answer. It most depends on what you care about, such as how quickly the development process, how easily the implementation, etc.
And which one is popular, which one is cool, I don't think it make really sense.
In my personal opinion, I'm good at ASP.NET and I can get Windows server easily, so I'll start with an ASP.NET service to provides data.
And, to be continued.
I am wondering if it would be possible to develop an enterprise-level web application without the use of a standard MVC structure and application server by carrying the business/flow logic and session data to the client-size Javascript and make it talk to REST data services directly...maybe we could make use of an authorization/authentication layer and a second validation layer sitting on top of the data services. All these services operate on standard HTTP methods, support configurable logging&monitoring, and content or query parameters are all contained in the HTTP request/response body. Static HTML and Javascript are served to the browser and the rest is carried out by Javascript functions talking to the HTTP-based authorization/authentication, validation and then data services. Do you think this kind of an architecture could satisfy enterprise-level web application requirements?
It's possible but unlikely; what are the drivers that suggest this architecture to you? Is it just to be different or are there some specific aspects that this best addresses?
by carrying the business/flow logic
and session data to the client-side
Javascript and make it talk to REST
data services directly
In theory you'd still be able to have an appropriately layered solution (Business Logic (BL) script vs UI focused script) but practically speaking it'd be messy, and you'd lose the ability to physically separate it into different tiers. This could "bite" you at any number of places in the life of the system.
"Enterprise" grade systems are seldom small, I hate to think how much logic you'd be having to send over the wire to support a given action/process.
Putting all the BL into a scripting language ties you to that platform, and platforms change over time. The bad thing about scripts is that whilst they are stable to a degree I'd suggest they are more exposed to change than server-based platforms like Java or .Net. In an enterprise scenario, the servers will have very tight change control and upgrade paths mapped out for them - whereas browsers are much more open to regular change.
There's the issue of compatibility - unless you're tied to a specific browser (to the version level) guaranteeing consistent behavior is going to be harder, and will likely require more development effort. Let's say you deliver the solution successfully; what do you do when the business wants to take advantage of mobile computing - say iPads? Your only option is going to be a browser - you won't be able to take advantage of any of the native advantages of the platform. "The web and browsers" might seem like they'll be around forever - but then I'm guessing that what MainFrame folks said at the time. A server-centered solution is going to give you more life for less expense.
Staffing will be an issue - you'll need very strong JavaScript and server-side developers.
Security: having your core BL out on the client where it's much more exposed sounds very dangerous.
EDIT:
Web Apps can be sow for many reasons - not many of which are reason enough to put all your BL in JavaScript on the client. Building apps for performance is a whole field of endeavor on its own - I suggest you get more familiar with Architecting and implementing for performance before you write off n-tier web apps altogether :)
Regarding keeping your layers separated: there are different ways of doing this but it boils down to abstraction - and more correctly to keeping good design principles in mind; if you haven't heard of SOLID that would be a good place to start. In terms of implementation start reading up on Dependency Inversion (FYI - self-promotion, the articles mine and is .Net focused, but you should have no problem tracking down Java-based ones too).
Can you give examples how you used gSOAP and how well it was integrated in your existing architecture? Have you found development bottlenecks with gSOAP?
We used gSOAP for a bunch of ARM clients to communicate with an AXIS Web Service server. Pros of gSOAP:
very powerful, supports nearly all Web Service constructs
easy to use, its abstraction of WS calls into functions removes all Web Service complexity to the programmer
elegant interfaces in both C and C++
However, we ran into several development bottlenecks:
when using custom datatypes like maps or sets, it takes quite some hacking to get the gSOAP compiler to handle them (marshal/unmarshalling). Especially bad with dynamic data structures.
debugging is hard because of its intrinsic complex network, parsing and memory allocation parts. Do everything possible to stick with static memory allocation.
the mailing list is alive, but developers are not very active on it. Simple questions can get answered quickly, but the toughest problems often go unanswered
forget about optimization. Linking in gSOAP eats about 1MB of memory at runtime (-Os). Runtime performance is fine on our 32MB linux-based ARM board, but there really is little to do about optimization if you need it.
We used gSOAP in a C++-based web server about 4 years back. Overall it worked fine. The only major issue was that the interface was in C and procedural (I understand it is difficult to design a good non-procedural interface). There may be a lot of repeated code while implementing the interface, for which you might have to use macros (we didn't explore the templates option too far then).
We are using gSoap to deploy a web service onto an embedde linux device running an ARM MX processor.
We are using gSOAP to consume a WCF based webservice from an application deployed on a linux device running on ARM processor. The experience is good to a large extent.
We used gSOAP in a web server on ARM ARM9 400MHz device.
gSOAP daemon connected to a database daemon through zeromq library, which is run on the same device.
It supports more than 1000 basic requests wich does not requre connection to database.
Disabling support for multi-referenced SOAP option by the WITH_NOIDREF define helped to decrease serialization time about 4 times faster on big requests with large number of serialization nodes.
If I have make an application, how would I make it interact with a database on a server of mine? I'm kind of lost, and don't know where to start.
Vague question yields a vague answer.
It entirely depends on what kind of interaction you need. Lots of constant queries? Few and far between? Client side cache? Real time updates? All of these questions will impact the answer.
The easiest way is to go with an AJAX style HTTP based client/server type of interaction. Sticking a database behind a web server has been done about a bazillion times and, thus, you'll find lots and lots of examples and, even, solutions with a few google searches.
You don't really need to use JavaScript (the J in AJAX). Instead, send over an HTTP request that encapsulates your query and have the server respond with an XML document containing the answer.
If that won't work for you -- too much overhead, need to relay binary information (for which XML sucks), etc.. -- then you'll wan to go with more direct access to the database. That is harder and can range from porting a client library to the iPhone to creating your own wire protocol.
A significantly harder problem and you'll have to deal with networking issues -- firewalls, NAT, proxies, etc... -- that are generally already solved with HTTP.
Stick with HTTP until you prove that it won't work. Much simpler.
Search for "http request" in the iPhone docs. HTTP client APIs are included.
You might try Apple's "Getting Started with Networking & Internet" or "Introduction to the URL Loading System", in addition to the Networking section of the iPhone Application Programming Guide. There are also several sample applications that handle networking of various types within the ADC's iPhone section.
Additionally, there are a number of open source iPhone applications out there that handle networking, including my own.
I had a similar question regarding a rails app - the answers there may help you.
What is the best approach for building an iphone client for a rails app?
But the answer really depends on your knowledge of iphone programming and server side programming, plus how your database is set up at the moment - but most likely you're going to need to write some kind of webservice / REST API that allows a remote client to do things with your database.
There are many frameworks available for that on the server side - if you're starting from scratch, ruby on rails may be a good choice.
On the iphone side, you'll probably want to start by reading up on NSURLConnection, and various request/response formats that you can use with it. I've found exchanging plists between the phone and server to be a pretty easy approach.