A friend has asked if I could implement a data historian. I am busy doing research, googling around, reading UPC Unified Architecture - but it's a lot to get through, so I will ask if anyone here has ever gone down that road (while still continuing my research).
Approx how many man months for a 20+ year developer (or two) to get at least a demonstrable working prototype - and how long to completion?
Which programming language? Is C++ good, or what?
What resources are available to me? (I thought I saw an Open OPC framework, but can't find it again). Any FOSS, libraries or free code which I can base upon? Maybe a sourceForge project?
How best to test?
Any other hints?
In my company we use mostly to brand of Historian, PI developped by OSISoft and GE Profecy Historian. Ge Profecy now offers a 25 tag version of their latest Historian 4.5. The way it works is you got an historian server that collects from data collector pc's. Depending what piece of equipment you are communicating with you'll need different OPC driver.
Matrikon and Kepware are the 2 references in that field. At Matrikon you'll find almost anything related to OPC. We mostly use Kepware, cause we felt their solutions are more stable on the long run.
Depending of your knowledge of the PLC's you have in place and the number of point you want to acquire. It might take a day to a week implementing an historian. I'll be more than happy to help you if you provide us with more details.
It would be interesting if you can do a write up of your project when you complete it.
For OPC libraries your pretty limited, but OPC Connect has a good list of UA development kits otherwise you'll need to be a corporate member with the OPC Foundation.
This is an old topic, but I'm interested in the subject.
There is a Python library for OPC: openopc at SourceForge.net (I use a proprietary OPC client because it is provided by my automation supplier, Yokogawa.)
For short-term data grabs you can use a delimited text file, but for a historian you ought to use a database. I use SQLite for speed, size, and portability. Other DB solutions have advantages. Of course if you collect 400 points every second then over time your DB grows quite quickly, so efficient data storage is important.
Language used is influenced by your choice of OPC package. OpenOPC for Python is, well yes, for Python. I've used Graybox's free OPC client with .Net. The OCX I use at work is easiest to use with VB6. Not sure about others.
The time required to build a historian depends entirely on how complete the application needs to be. You can probably put together a data grabber in a few hours. A long-term historian with interfaces to view data, to add and delete points, to maintain data integrity, to handle bad data and interrupted communications gracefully -- all that will take days, not hours.
Related
I'm wondering if for simply communicating with a PLC, like reading and writing tags, do I need all of the other heavy lifting that comes with an OPC-UA server?
I've tried writing a simple server in Python that talks to the PLC, but I get denied when requesting information from the PLC.
The Controllogix PLC I'm attempting to communicate with uses Ethernet/IP to communicate, so why doesn't a simple server/client script work? What is required exactly to communicate with an Allen Bradley PLC or PLC's in general?
There is quite a bit required to communicate with a PLC.
Each vendor has a driver, there are firmware compatibility considerations. Different protocols to think about.
OPC-UA makes it a bit more generic, but OPC-UA still has a set of things to work around when setting up communications.
Most of the OPC products I've worked with, needs to have their security adjusted to allow anonymous communication. It's generally bad practice to do this. (A network intrusion would be able to read/write to your automation layer) There is certificate signing and some encryption business that needs to be turned off if you're looking for simple communication. (Again, not a good practice but ok for learning)
After all that you have to have a notion of how your PLC is set up on your OPC server, there are channels, devices, namespaces etc. You'll point the OPC client to some opc.tcp://:
If you got this far you're almost done, I'm assuming your OPC server is running and has tags configured at this point. You can use your OPC-UA API to do a read. It can return just the value, or you can get an object back with tag health, timestamp, and a bunch of other data. Depends on the implementation. After that you can do subscriptions, writes...whatever else you need.
TLDR: OPC server not required, but may be the easiest method. Turn off security. (But turn it back on before exposing your control layer to the net)
I am also a little late to this conversation. If you are interested in coding your own solutions and don't want to use any of the commercially available standards, AdvancedHMI is a "mostly" open source solution written in VB.NET which is 100% free and provides communications to many different PLCs including the ControlLogix platform. Since I see you are programming in Python you may also be interested to know that the project does work under Mono on the Linux OS. I have used it to write gateways between EthernetIP and ModbusTCP and to pull data serially from OEM devices and push this data to a CLX PLC.
The forum is full of many helpful hints and is very active and supported.
Just trying to give you another option. DDE, NetDDE, FastDDE, OPC, DCOM, Suitelink.... These are all good, but mostly a pay to play adventure. As a programmer, it seems ridiculous to have to pay such an excessive amount of money just to talk to my hardware, IMHO. Sorry for the rant. Have Fun!
Update - Just wanted to also suggest the following open source project written in python:
https://github.com/dmroeder/pylogix
I have used this to write small programs for communicating with CompactLogix and ControlLogix. (Even to/from a RaspberryPi!)
Depends on several factors, if you want something simple to program you can opt for Modbus/TCP I think some AB PLC supports it without extra hardware.
However if you want something with more security for example for industrial use then OPC UA would be better choice but the programming has a complexity far higher than Modbus, even using the libraries of OPC Foundation or others. There is the option of using a commercial or free (if any) OPC UA server to save work, then you will need to program the client side only.
With Ethernet/IP it should also be possible, but the problem is that there is no clear specification and even different AB models talk different Ethernet/IP dialect ! , it is also far more complex to program than Modbus.
I am a little late to this discussion, but there are a couple commercial tools that make this a bit easier. The one that comes to mind for me when you say you are using python is Cogent's data hub. It is certainly not the cheapest tool out there, but they have already done all of the heavy lifting for PLC/PC communications & security.
If trying to read CLX data using Python, there are several open source implementations that will save you a lot of work. Such as this:
https://github.com/dmroeder/pylogix
If you use .NET and Visual Studio, you can use AdvancedHMI
to be able to read and write OPC Tags to ControlLogix platform is done via its communication Driver RSLinx. RSLinx acts as an OPC Server, it will need to be configured to communicate with the PLC and run on a networked PC on the same LAN. Several flavours of RSLinx are available (for WAN/VLAN also) but essentially this is the communications driver you need to talk to AB PLC's
I have an application whereby I need to read data from a PLC into a database and whereby I need to develop my own application to do this. I just need to read 5 values from the PLC and log it to a DB. I have a demo OPC server running and can access it via UA or DA.
After looking at many different approaches I settled on using an OPC server to connect to the PLC and then to write an OPC client to interface to the OPC server and then write the data to the DB from my app. My language of choice is C# with .Net and the only licence fee I am able to pay for is for the OPC server from my PLC vendor.
What I am however finding extremely frustrating is getting the right info on OPC to get started. I dont want to buy any stacks but would prefer an open source stack. The information seems very fragmented and all over the place. Most of the info about OPC seems to be hype about how easy it is to use etc.
The best post on Stackoverflow that I could find is: Noob guide to OPC: how to write a C# Hello World client? and some of the links are not active any more.
My question is therefore are there any good tutorials showing how to build and OPC client from scratch in .net and what is the best open source SDK to use without having to buy a vendor's stack?
Is DA also worth learning or should one stick with UA?
The big question is why is OPC so frustrating when its marketed as being so easy?
It would also be nice to have a nice high level guide on the theory that one needs to know to build a client. I do realize that with time its possible to eventually figure this from the resources available but with limited time to make sense of all the scattered resources a quicker guide would be helpful.
Stick with OPC UA.
Luckily for you, the OPC Foundation's C# reference implementation has the capabilities of both stack and SDK, whereas other language reference implementations are typically just stack functionality.
The code is available on GitHub: http://opcfoundation.github.io/UA-.NET/
If you're not a member of the foundation the code is available under GPL2.
As for your concerns about the ease of use and marketing... I assume it's because OPC UA is marketed towards end users, who will just be hooking up various OPC UA compliant applications, which is easy. As a developer I think its fair to say there's a little more assumed about your ability to figure things out... from code examples, the specifications, books that are available, etc...
I'm actually trying to find the best way to write a simple XMPP server or find one that leaves a really small foot print. I only care about the core features of XMPP (statusing, messaging, group messaging) . Also learning about XMPP protocol at the moment, so please bear with me if i'm saying something that is naive.
So currently, open fire is being used in our application. It's really heavy and has a lot of features that aren't necessary for our needs.
I've looked at tigase which is another XMPP server that leaves a smaller footprint.
I also looked at apachi mia framework which contains or has an extension module (vysper - XMPP server). I've tried to google reviews on this when compared to other servers, but can't find that much information.
In general, I"m looking for the best way to implement a simple XMPP server either from scratch or from an already built library/framework in java which leaves a smaller foot print then open fire. Interested in getting more pro/cons with vysper in comparison to openfire and tigase as well.
Any advice appreciated,
Thanks,
D
I can talk for Tigase only as I am Tigase team member.
For basic XMPP features Tigase binaries take no more than 1.5MB and if necessary it could be reduced to something below 1MB.
We run periodic tests with small memory and all the basic features work fine even if you run Tigase with 10MB of RAM, of course for no more then a few dozen online users.
I think this is quite small foot print. We put a lot of effort on optimizing and testing. It is also designed to easily integrate with other systems so this is the way to go :-)
Of course I am biased towards Tigase which is understandable. Please wait for some more independent opinion.
There are must be a lot of apps that are designed to communicate with server. My question is only about App installed on iOS device + Server side service interaction. Web app is not what I am talking about, and there should be no webpage involved in this discussion at all. Typical examples are Apps like Instagram and Twitter, in which most of the information exchanged between the App and the server is just data like String, Image and Integers(wrapped in JSON or XML), no webpage presentation needed.
My question will be: if you are an independent app developer, and you are designing such an app from scratch without any existing website API, database structure or application(so you are not limited by any existing API or database structure or application protocol), What will be the most efficient approach?
What the sever side need to do are:
receive data send by the App;
process the data with designed logic;
interact with database(like MySQL);
do necessary data mining and analysis---this could be a constantly running service or one time task requested by the App client;
send the data back to the App upon request or spontaneously;
exchange or broadcast the data between/among different App clients (i.e.: group chatroom and peer to peer message);
As far I as know there are 3 obvious options to implement the server side:
PHP
Python
Ruby on Rails
(please feel free to add more options)
My questions are:
which one is the most appropriate choice to implement the server side?
If the App is focusing intensively on natural human language/text searching, analyzing and data mining, which one is the best choice? I heard Python is doing pretty good in this area.
Any advice on the database choices? I am using MySQL for now, and I found it's quite powerful for my purposes, I heard Twitter is switching to Cassandra. Will that be too difficult to start with?
For the server end, if you need to build a Server management interface, for you as an admin to manage and monitor the community, membership, data and such, is there any existing solution, or framework or tool for that? what will be the most efficient approach?
If a new programmer has no experience in non of them, which one you suggest he/she to start with?
Is there any good reference material or sample code on the server side in such context we can learn from?
I know there are a lot of very experienced experts on these areas on Stackoverflow, but I saw more newbies who just entered the iOS developing area without much knowledge in server/database programming experience. And I hope this thread can help these who are thinking to design an App with server/client structure but have no idea where to start with.
ps: I will keep updating this question thread and adding my findings on this topic, to help all other users at stackoverflow. :-) Please try to make your answer informative, easy to understand, and constructive. I guess most of readers for this thread will be new members of this great community.
Are you sure you want to spend time & money to develop your own Server & develop your own API?
There are lots of mBaaS (mobile Backend as a Service) providers today such QuickBlox, Parse,StackMob, which are ready to use and they have great Custom Objects API and some of predefined modules. They have great free plans with big quota. Some of them such QuickBlox has Enterprise plan - so you can buy license and they server team update server for you purpose.
So, i recommend not develop your server and think about mBaaS market.
Just about your issue - I can recommend look at QuickBlox Custom Objects code sample and also Custom Objects API. Custom Objects module provides flexibility to define any data structure(schema) you need. Schema is defined in Administration Panel. The schema is called Class and contains field names and their type. I think it's what you need.
which one is the most appropriate choice to implement the server
side?
Well that depends on what you know, there is reason to choice one of the other
If the App is focusing intensively on natural human language/text
searching, analyzing and data mining, which one is the best choice? I
heard Python is doing pretty good in this area.
This would reflect on your first question, you pick the language on you needs. Thus if python makes it easier then pick that one.
Any advice on the database choices? I am using MySQL for now, and I
found it's quite powerful for my purposes, I heard Twitter is
switching to Cassandra. Will that be too difficult to start with?
Again not one that is easy to answer, since it all has to do with requirements. But any SQL server will do. Cassandra is meant for "scalability and high availability without compromising performance" accourding to there website. Do you think you webservice will get many request then it might be a choice to consider.
For the server end, if you need to build a Server management
interface, for you as an admin to manage and monitor the community,
membership, data and such, is there any existing solution, or
framework or tool for that? what will be the most efficient approach?
This again is only going to be answered when you pick the a SQL server and server language.
If a new programmer has no experience in non of them, which one you
suggest he/she to start with?
Start with something simpler, you are really going out on a limb here.
Is there any good reference material or sample code on the server
side in such context we can learn from?
Propably there is some, but you should really start small and work from there.
Twitter started out as a Rub on Rails app and is working on scalability and availability which ruby is not really good ar (that is my person opinion). or Look at facebook they have written a php to c compiler to make php run faster.
The only thing I can say to start code, when you app does take off then tackle the some of the performance issues.
And since you state that you are new to programming do not bite of more then you can chew.
This is a huge question and I don't think there is a best answer. It most depends on what you care about, such as how quickly the development process, how easily the implementation, etc.
And which one is popular, which one is cool, I don't think it make really sense.
In my personal opinion, I'm good at ASP.NET and I can get Windows server easily, so I'll start with an ASP.NET service to provides data.
And, to be continued.
Good day,
Our school, a small high school in semi-rural New Zealand, is currently looking into online homework solutions. Being one of the IT guys, I have been asked to look into some of the options. We have checked around and there are no robust solutions that cover what we are looking for. So, we are considering development of our own system, either on our own or in collaboration with some other schools.
Before I put significant time into any one option, I would thought I should ask for some expert advice.
Please keep in mind that one of our major obstacles is that around 20% of our students are on dial-up because broadband is not available in their area.
We are also not limited to the technologies listed, they just are the ones that we have been looking into up to this point.
With that in mind, here goes.
1. Is there a way to pre-determine the bandwidth needed for these technologies?
2. If bandwidth continued to be too limiting, could the final solution stand alone so we could distribute it to students on CD or USB stick?
3. What are some pros/cons of each for use with databases, specifically mysql or postgresql? (After all we do need to keep track of lots of data)
4. What are some pros/cons of each for of these RIA development?
I appreciate everyone for sharing their time and expertise on the matter.
Cheers,
Ben
1) If you write full-AJAX application, such as in GWT, the bandwitch will be:
a) the size of application java script, images, etc., you may consider that everything is loaded when user logs in (cache for images may seems to be big, but it's easily overloaded)
b) the size of communication - in GWT it depends only from you! no magic full-frame reloading, sending is only what YOU are wanting to send
2) I do not catch your point, stand alone applications can be distributed such way, applications that use databases generally can't
3) postgresql has high compatibility with Oracle - same transaction+select for update behaviour, pgPLSQL is highly inspired by PL/SQL (easy to rewrite stored procedures).
I personally suggest MySQL for a school project for its simplicity. PostgreSQL is powerful but a bit complicate to configure and the visual tool for optimizing queries not good.
Without considering the bandwidth, I definitely suggest ZK since, again, it is much easier to learn, to develop and to maintain (also much more powerful). The bandwidth consumption and latency of GWT really depends how much effort you want to invest, and how skillful your people are familiar with distributed computing, while the network bandwidth is basically the states of UI (not data), which is reasonably small. In short, you could have the best network bandwidth and latency if you optimize it at the best with GWT, while ZK is less to worry but, if you want to improve, you have to use jQuery (i.e, in JavaScript).
Thanks lechlukasz, I appreciate your comments and insight.
I will clarify my point about stand alone applications. We have a number of students, as high as 20%, who do not have access to broadband due to their geographic location. We are considering, as part of the design, how we may be able to distribute a stand alone version.
For instance, if we were to abstract all the database calls using a separate class in GWT, we could recompile a stand alone version that didn't make the database calls. The database would likely only be for tracking results and reporting.
In reality, we would likely implement the front end product first with references to empty methods for storing the results in a database and implement those methods at a later time.
For the record, we have started to code up some test cases using GWT/SmartGWT and are pleased with the results. Although we cannot comment on the other technologies considered because we didn't try them to the same extent, we are pleased with the results to this point of the project.
Cheers,
Ben