Recently, I created a lightweight wrapper for the C++ boost asio library for some network communication. I used it to prototype some new functionality. We quickly moved over to a system that utilized Kafka to take advantage of an existing microservice framework when more internal funding came our way. No problem, I figured we would move to a different network model later on, and the internals were more important to my job than the network communication.
My question is, with the amount of technologies that abstract away network interfaces, (i.e. Kafka, grpc, ActiveMQ, ZeroMQ, etc.) are the use of base TCP/UDP sockets becoming more of a last resort, where software architects try to find an existing broker/stream processor/network message passing tool to fit their model? Or are there still many new production developments utilizing base level TCP/UDP sockets, not including those who solely write network libraries such as those mentioned above?
Note that I don't work with Kafka, grpc etc. in my line of work, but I have used UDP/TCP sockets extensively in the past. So forgive any misunderstanding of those particular technologies.
Related
I want to dive into the whole diversity of tools which provide connection between programs over the network.
To clarify the question, I divide it on subquestions:
Why some groups of programs (or specific tools/frameworks/approaches with programming languages where this frameworks can be used) were popular in each period of time? (I expect description of problems which were solved, description of tools, why those tools are considered as best solution to those problems at that time, why some tools lost popularity)
What is the entire history of software communication over the network? (tools/approaches popularity precisely to decades)
What are the modern solutions to this problem?
I can distinguish only two significant approaches.
RPC, RMI and their implementations (I saw this, but it is about concrete problem and specific tools to solve this problem, I want to see the place of this problem in the whole picture of interconnection programs over the network. I heard about implementations: ONC RPC, XML-RPC, CORBA, DCOM, gRPC, but which are active now? which are reasonable to use? which are preferable and why? I want answers not to be opinion based, so I accept answers like "technology A better than technology B for problem X because ..." only if there is reliable research/statistics or facts). I heard that RPC and RMI were popular 10 years ago. Are they still?
Web services: REST, SOAP.
Am I miss something? Maybe there are some technologies which solve problem completely new way? Maybe there are technologies which can be treated as replacement to RPC(RMI) and Web Services? Can we replace RPC(RMI) by REST for any task? Can we replace RPC(RMI) by REST only for modern tasks? Should I separate technologies not as RPC and Web Services, but in some other manner?
As a partial answer, I can give you my feedback on the use of RabbitMQ.
As explain here, it provides a lot of different ways to use it :
RPC by implementing a "callback" queue
One to one, one to many routing strategy to propagate your events through your whole infrastructure and target the right destination.
It comes with the ability to persist messages to avoid loosing data when a crash appears but also with some plugins to increase possibilities (e.g x-delayed plugin)
This technologie written in Erlang is powerful and is a must try in term of communication between programs.
To your question „Am I missing something“: yes.
Very popular communication patterns are the so-called Event-Driven or Message-Driven protocols. This type of protocols are often used in distributed systems such web applications, microservices and IoT-Environments. The communication is complete asynchronously and allows building scalable and loosely coupled systems.
There are many different frameworks and methods for Event-Driven systems like WebSockets, WebHooks, Pub-Sub and Messaging-Librarys like AcitveMQ, OpenMQ, RabbitMQ, ZeroMQ and MQTT.
Hope this info helps for your research.
I'm wondering if for simply communicating with a PLC, like reading and writing tags, do I need all of the other heavy lifting that comes with an OPC-UA server?
I've tried writing a simple server in Python that talks to the PLC, but I get denied when requesting information from the PLC.
The Controllogix PLC I'm attempting to communicate with uses Ethernet/IP to communicate, so why doesn't a simple server/client script work? What is required exactly to communicate with an Allen Bradley PLC or PLC's in general?
There is quite a bit required to communicate with a PLC.
Each vendor has a driver, there are firmware compatibility considerations. Different protocols to think about.
OPC-UA makes it a bit more generic, but OPC-UA still has a set of things to work around when setting up communications.
Most of the OPC products I've worked with, needs to have their security adjusted to allow anonymous communication. It's generally bad practice to do this. (A network intrusion would be able to read/write to your automation layer) There is certificate signing and some encryption business that needs to be turned off if you're looking for simple communication. (Again, not a good practice but ok for learning)
After all that you have to have a notion of how your PLC is set up on your OPC server, there are channels, devices, namespaces etc. You'll point the OPC client to some opc.tcp://:
If you got this far you're almost done, I'm assuming your OPC server is running and has tags configured at this point. You can use your OPC-UA API to do a read. It can return just the value, or you can get an object back with tag health, timestamp, and a bunch of other data. Depends on the implementation. After that you can do subscriptions, writes...whatever else you need.
TLDR: OPC server not required, but may be the easiest method. Turn off security. (But turn it back on before exposing your control layer to the net)
I am also a little late to this conversation. If you are interested in coding your own solutions and don't want to use any of the commercially available standards, AdvancedHMI is a "mostly" open source solution written in VB.NET which is 100% free and provides communications to many different PLCs including the ControlLogix platform. Since I see you are programming in Python you may also be interested to know that the project does work under Mono on the Linux OS. I have used it to write gateways between EthernetIP and ModbusTCP and to pull data serially from OEM devices and push this data to a CLX PLC.
The forum is full of many helpful hints and is very active and supported.
Just trying to give you another option. DDE, NetDDE, FastDDE, OPC, DCOM, Suitelink.... These are all good, but mostly a pay to play adventure. As a programmer, it seems ridiculous to have to pay such an excessive amount of money just to talk to my hardware, IMHO. Sorry for the rant. Have Fun!
Update - Just wanted to also suggest the following open source project written in python:
https://github.com/dmroeder/pylogix
I have used this to write small programs for communicating with CompactLogix and ControlLogix. (Even to/from a RaspberryPi!)
Depends on several factors, if you want something simple to program you can opt for Modbus/TCP I think some AB PLC supports it without extra hardware.
However if you want something with more security for example for industrial use then OPC UA would be better choice but the programming has a complexity far higher than Modbus, even using the libraries of OPC Foundation or others. There is the option of using a commercial or free (if any) OPC UA server to save work, then you will need to program the client side only.
With Ethernet/IP it should also be possible, but the problem is that there is no clear specification and even different AB models talk different Ethernet/IP dialect ! , it is also far more complex to program than Modbus.
I am a little late to this discussion, but there are a couple commercial tools that make this a bit easier. The one that comes to mind for me when you say you are using python is Cogent's data hub. It is certainly not the cheapest tool out there, but they have already done all of the heavy lifting for PLC/PC communications & security.
If trying to read CLX data using Python, there are several open source implementations that will save you a lot of work. Such as this:
https://github.com/dmroeder/pylogix
If you use .NET and Visual Studio, you can use AdvancedHMI
to be able to read and write OPC Tags to ControlLogix platform is done via its communication Driver RSLinx. RSLinx acts as an OPC Server, it will need to be configured to communicate with the PLC and run on a networked PC on the same LAN. Several flavours of RSLinx are available (for WAN/VLAN also) but essentially this is the communications driver you need to talk to AB PLC's
What I want to do: I want to add communication capabilities to a couple of applications (soon to be jar libraries for Java) in Scala, and I want to do it in the most painless way, with no Tomcat, wars, paths for GET requests, RPC servers, etc.
What I have done: I've been checking a number of libraries, like Jetty, JAX-RS, Jackson, etc. But then I see the examples and they usually involve many different folders for configuration, WSDL files, etc. Most of the examples lack a main method and I don't have a clear picture about how many additional requirements may they have (e.g. Tomcat).
What I am planning to do: I'm considering to simply open a socket on the "server" to listen, then connect with the "client" and transfer some JSON, in both directions. This should be fairly standard so that I can use other programming languages in compatible ways (e.g. Python).
What I am asking: I would like to know whether there is some library that makes this easier. Not necessarily using raw sockets, but setting up some process communication in just a few lines, maybe not as simple as Node.js, but something similar.
Bonus: It would be cool to
be able to use other programming languages (e.g. Python) by using open standards
have authentication
But I don't really need any of those at this point.
I think you need RPC client/server system, I would suggest to take one of these two:
Finagle - super flexible and powerful RPC client/server from Finagle. You can define your service with Thrift, and it will generate stubs for client/server in scala. With Thrift it should be straightforward to add Python support.
Spray - much smaller library, focused on creating REST services. It's not so powerful as Finagle, however much easier. And REST allows you to use any other clients
Remotely - an elegant RPC system for reasonable people. Interesting and very promising project, however maybe difficult to start with because of extensive Scalaz+Shapeless+Macro usage
Honestly if you want something that is cross-language compatible, simple, straightforward, and concise then you do not want to use plain old sockets!
Check out dropwizard. It is amazing and I use it for small and large projects alike! It is usually configured by no more than a single configuration file. It supports authentication too!
Out of the box it gives you really great inter-process communication over JSON (using Jackson) and much much more. There is also pretty decent Scala support for dropwizard.
If you must roll your own then I'd recommend using Jackson for JSON parsing. It's super simple to use and also has great scala support.
If you've got a "controlled" use case where the client and server are on the same LAN and deployed in tandem, I'd (controversially) recommend Java RMI; it's dumb and JVM-specific (and uses a Java-specific protocol), but it's very simple to use.
If you need something more robust and cross-language, I'd recommend Apache Thrift. You write your interfaces in a platform-independent interface definition language, and it's very clear which changes are compatible and which are not; the thrift compiler generates skeleton interfaces for you to use, and then you just write an implementation of that interface and a couple of lines to start the server (as you can see from the example on the homepage). It's also got good support for async implementations if you need the performance. Thrift itself is reasonably standard and cross-platform, with its own binary protocol, or you can use JSON as a transport if you really want to (I'd recommend against that though).
RabbitMQ provides one easy way to do what you want without writing a server and implementing your own persistence, flow control, authentication, etc. You can brew or apt-get install it.
You start up a broker daemon process (i.e. manages message queues)
In the Scala producer, you can use Maven-provided Java API to send JSON strings without any fuss (e.g. no definition languages) to specified queues
Then in your other Scala program, connect to the broker, and listen for messages on the queue, and parse the incoming JSON
Because it is so popular, there are many tutorials online for different patterns you may want to use to distribute the messages, e.g. pub/sub, one-to-one, exactly-once delivery, etc.
I need to implement a distributed XMPP MuC application on the lines of XEP-0289 minus some of the features, in essence I want to have a bare bones implementation of the plugin, my concern is to address fault-tolerance and as of now I do not want to worry about the performance considerations as specified in 289.
I have looked into SleekXmpp as a tool to develop server side plugins, but don't know how comfortable it would be to use it for such an implementation, other options I have looked at are OpenFire , Tigase. I am comfortable with Python/Java and other key features to consider would be good documentation, ease of use etc keeping that in mind I would like to know what would be the preferred path to take for this development.
Any guidance will be appreciated.
you should be able to write a MUC component that includes FMUC (or similar). The general way to do this would be to use a library that supports XEP-0114 components (e.g. SleekXMPP (Python), Swiften (C++)) and implement MUC+FMUC through that. You haven't said what your concerns with SleekXMPP are, but it's a fairly well-respected library in the XMPP community, so seems a fair choice (I'd pick Swiften, but I'm biased as one of the authors).
Your second option (patching the server directly) isn't generally the XMPPish way of adding customisations (as it's vendor-specific), but should also work if you can find someone sufficiently familiar with the server code, or if you're willing to become so.
To achieve fault tolerance (assuming you mean resilience to server failures) you'd need to run your XMPP server clustered, and also cluster your FMUC implementation. With that done, the usual XMPP fail-over using SRV records in DNS should ensure other servers retry connections to another host.
On a side note, the next version of FMUC (XEP-0289) will have some of the features of the current revision stripped out, and a number of improvements made based on deployment experience, so if your work is not time-critical, it might be of benefit to you to read that when it's released. I also note that there exists at least one implementation of FMUC already (Isode's M-Link, on which I work), and there is interest from other vendors, so using the standard protocol might benefit you in terms of not re-inventing the wheel.
Is there any known effort to build a generic software framework/library for building distributed/P2P systems on the wider internet[1]? Something that ideally should provide basic services such as peer autodiscovery, autoconfiguration (wrt NAT/firewall issues, bootstrapping), fault-tolerance (churn-resistance, adaptivity, replication), differentiated transports (reliable, unreliable, unicast, anycast, multicast, broadcast), basic storage (DHT-like) and events notification (pubsub).
So, basically something like Hazelcast but not limited to walled-garden server-to-server applications. This would be tremendously useful to easily build distributed applications.
[1] with "wider internet" I mean the public internet, including mobile, residential, eventually connected and possibly byzantine endpoints
you will find different option and technologies depending on the framework you want to use.
For instance for Java based applications you might want to have a look at the JXTA technology.