I spotted a very good argument against protocol-based (viz. SOAP) reliable messaging. Is this a flame-war topic, or is there a fair degree of consensus about this?
I think the author's "Netherlands" case-study ought to have included an in-order example as well.
p.s. I wish google had a "contrast" tool, that finds sites with opposing points of view. Doesn't "Google Contrast" sound cool? :)
It's not an argument against reliable messaging in principle, but against putting it in the transport layer. It reminds me a little of the Mysql developers years ago saying "you don't need $foo in the database, it's easy to do that in the application layer" - where $foo was typically "transactions", or "foreign keys", or "constraints", or ... any of many other things they've since decided to implement anyway.
I don't mean that to imply that de Graauw is necessarily wrong, as it's clear he has no need for a reliable transport in his particular app, but I do suggest that "I don't need X, therefore you don't either" is a false generalisation in principle and often incorrect in practice.
Unreliable messaging (ie. UDP) can be made reliable by providing handshaking messages:
S: Are you there?
R: Yes, I am Here.
S: Turn On Porch Light (this is Message 6)
R: OK.
S: Did you receive message 6?
R: Sorry, what? (or no response)
S: Turn on Porch light (this is message 7)
R: Message 7 Received.
R: Porch Light is On.
Related
I understand an interface is a set of publicly exposed things that one system can use to interact with others systems. I'm reading about WEBRTC protocol and to understand what a protocol is I went to the Wikipedia definition. It says more or less that a protocol is a system of rules that allows two systems to communicate. Ain't that the same as interface? Maybe I'm not understanding one or both.
An interface defines how two entities may communicate. A protocol defines how they should communicate and what that communication means.
Here is an interface:
public interface ICommunicate
{
string SendMessageAndGetResponse(string message);
}
A protocol then might be:
Send "Hello", if you get back "Hi" then send "How are you?" and the response will be a status. If you get back anything other than "Hi" from the initial message then the system is not functioning properly and you must send the message "Reboot" which you'll then get back "Rebooted!" if successful and anything else for failure.
In general interface mean "The point of interconnection or contact between entities." and transferred to software it means "The connection between parts of software." and also "In object-oriented programming, a piece of code defining a set of operations that other code must implement." (Source)
In general protocol means "The official formulas which appeared at the beginning or end of certain official documents such as charters, papal bulls etc." and transferred to computers it means "A set of formal rules describing how to transmit or exchange data, especially across a network.". (Source)
So protocol focuses more on the data exchange, whereas interface focuses more on software interaction independent of any data exchange.
Of course, in the end, software interaction is most of the time a data exchange. Passing arguments to a function is a data exchange. Calling a method/function is not directly a data exchange but you need to imagine it like this: Instead of calling different functions:
c = add(a, b);
c = sub(a, b);
you could as well always call the same function and pass the desired functionality as argument:
c = func("add", a, b);
c = func("sub", a, b);
and that way the functionality becomes data as well.
The terms are somewhat interchangeable. E.g. some programming languages call it interface to focus on the pure interaction of components (classes, objects, etc.) and some call it protocol to focus on the data exchange between the components.
On a network, a protocol is how data is exchanged; think of IP protocol or TCP protocol. But if you have communication endpoints you talk to over a network to trigger functionality, e.g. a REST API, than the sum of all these endpoints and their parameters can be called an interface, while triggering one of the interface functions would be done via a HTTP request and HTTP is a protocol and defines how data is transferred.
I think in some ways the word 'interface' could also be used for this (especially as the I in API), but generally when talking about what we're sending over the wire, the common word is protocol.
When you dive deep enough into exact definitions of words, the meaning and difference can sometimes break down a bit.
But avoiding super exact semantics, API/Interface tends to be a bit higher level than protocol.
ZIO (https://zio.dev/) is a scala framework which has at its core the ZIO[R, E, A] datastructure and its site gives the following information for the three parameters:
ZIO
The ZIO[R, E, A] data type has three type parameters:
R - Environment Type. The effect requires an environment of type R. If this type parameter is Any, it means the effect has no
requirements, because you can run the effect with any value (for
example, the unit value ()).
E - Failure Type. The effect may fail with a value of type E. Some applications will use Throwable. If this type parameter is
Nothing, it means the effect cannot fail, because there are no
values of type Nothing.
A - Success Type. The effect may succeed with a value of type A. If this type parameter is Unit, it means the effect produces no
useful information, while if it is Nothing, it means the effect runs
forever (or until failure).
It's easy to get what A is: it's the value returned by the function in the nominal case, ie why we coded the function for.
R is so kind of dependency injection - an interesting topic, but we can just ignore it to use ZIO by alway setting it to Any (and there is actually a IO[E, A] = ZIO[Any, E, A] alias in the lib).
So, it remains the E type, which is for error (the famous error channel). I roughtly get that IO[E, A] is kind of Either[E, A], but deals with effect (which is great).
My question is: why should I use an error channel EVERYWHERE in my application, and how can I decide what should go in the error channel?
1/ Why effect management with an error channel?
As a developper, one of your hardest task is to decide what is an error and what is not in your application - or more preciselly, to discover failure modes: what the nominal path (ie the goal of that code), what is an expected error that can be dealt with by the application in some way later on, and what are unexpected errors that the application can't deal with. There is no definitive answer for that question, it depends of the application and context, and so it's you, the developper, who needs to decide.
But the hardest task is to build an application that keeps its promises (your promises, since you chose what is an error and what is the nominal path) and that is not surprising so that users, administrators, and dev - including the futur you in two weeks - know what the code do in most cases without having to guess and have agency to adapt to that behavior, including to respond to errors.
This is hard, and you need a systematic process to deals with all the possible cases without going made.
The error channel in IO bi-monad (and thus ZIO) helps you for that task: the IO monad helps you keep track of effects, which are the source of most errors, and the error channel makes explicit what are the possible error cases, and so other parts of the application have agency to deal with them if they can. You will be able to manage your effects in a pure, consistant, composable way with explicit failure modes.
Moreover, in the case of ZIO, you can easely import non-pure code like legacy java extremelly easily:
val pure = ZIO.effect(someJavaCodeThrowingException)
2/ How do I choose what is an error?
So, the error channel provide a way to encode answer to what if? question to futur dev working on that code. "What if database is down?" "there's a DatabaseConnectionError".
But all what if are not alike for YOUR use case, for CURRENT application level. "What if user is not found?" - ah, it may be a totally expected answer at the low, "repository" level (like a "find" that didn't find anything), or it can be an error at an other level (like when you are in the process of authenticating an user, it should really be there). On the first case, you will likely not use the error channel: it's the nominal path, sometimes you don't find things. And in the second case, you will likelly use the error channel (UserNotFoundError).
So as we said, errors in error channel are typically for what if question that you may want to deal with in the application, just not at that function level. The first example of DatabaseConnectionError may be catch higher in the app and lead to an user message like "please try again" and a notification email to sysadmin ("quick, get a look, something if wrong here"). The UserNotFoundError will likely be managed as an error message for the user in the login form, something like "bad login or password, try again or recover credentials with that process".
So these cases (nominal and expected errors) are the easy parts. But there are some what if questions that your application, whatever the level, has no clue how to answer. "What if I get a memory exception when I try to allocate that object?" I don't have any clue, and actually, even if I had a clue, that's out of the scope of the things that I want to deal with for that application. So these errors DON'T go in the error channel. We call them failure and we crash the application when they happens, because it's likely that the application is now in an unknow, dangerous, zombie state.
Again, that choice (nominal path/error channel/failure) is your choice: two applications can make different choices. For example, a one-time-data-processing-app-then-discard-it will likelly treat all non-nominal paths as failures. There is a dev to catch the case in realtime and decide if it's important (see: Shell, Python, and any scripting where that strategy is heavely used - ok, sometimes even when there is no dev to catch errors:). On the other end of the specter, Nasa dev put EVERYTHING in the error channel(+), even memory CORRUPTION. Because it is an expected error, so the application need to know how to deal with that and continue.
(+)NOTE: AFAIK they don't use zio (for now), but the decision process about what is an error is the same, even in C.
To go further, I (#fanf42) gave a talk at Scala.io conference. The talk, "Ssytematic error management in application", is available in French here. Yes, French, I know - but slides are available in English here! And you can ping me (see contact info near the end of slide deck).
Suppose I send objects of the following type from GWT client to server through RPC. The objects get stored to a database.
public class MyData2Server implements Serializable
{
private String myDataStr;
public String getMyDataStr() { return myDataStr; }
public void setMyDataStr(String newVal) { myDataStr = newVal; }
}
On the client side, I constrain the field myDataStr to be say 20 character max.
I have been reading on web-application security. If I learned something it is client data should not be trusted. Server should then check the data. So I feel like I ought to check on the server that my field is indeed not longer than 20 characters otherwise I would abort the request since I know it must be an attack attempt (assuming no bug on the client side of course).
So my questions are:
How important is it to actually check on the server side my field is not longer than 20 characters? I mean what are the chances/risks of an attack and how bad could the consequences be? From what I have read, it looks like it could go as far as bringing the server down through overflow and denial of service, but not being a security expert, I could be mis-interpreting.
Assuming I would not be wasting my time doing the field-size check on the server, how should one accomplish it? I seem to recall reading (sorry I no longer have the reference) that a naive check like
if (myData2ServerObject.getMyDataStr().length() > 20) throw new MyException();
is not the right way. Instead one would need to define (or override?) the method readObject(), something like in here. If so, again how should one do it within the context of an RPC call?
Thank you in advance.
How important is it to actually check on the server side my field is not longer than 20 characters?
It's 100% important, except maybe if you can trust the end-user 100% (e. g. some internal apps).
I mean what are the chances
Generally: Increasing. The exact proability can only be answered for your concrete scenario individually (i. e. no one here will be able to tell you, though I would also be interested in general statistics). What I can say is, that tampering is trivially easy. It can be done in the JavaScript code (e. g. using Chrome's built-in dev tools debugger) or by editing the clearly visible HTTP request data.
/risks of an attack and how bad could the consequences be?
The risks can vary. The most direct risk can be evaluated by thinking: "What could you store and do, if you can set any field of any GWT-serializable object to any value?" This is not only about exceeding the size, but maybe tampering with the user ID etc.
From what I have read, it looks like it could go as far as bringing the server down through overflow and denial of service, but not being a security expert, I could be mis-interpreting.
This is yet another level to deal with, and cannot be addressed with server side validation within the GWT RPC method implementation.
Instead one would need to define (or override?) the method readObject(), something like in here.
I don't think that's a good approach. It tries to accomplish two things, but can do neither of them very well. There are two kinds of checks on the server side that must be done:
On a low level, when the bytes come in (before they are converted by RemoteServiceServlet to a Java Object). This needs to be dealt with on every server, not only with GWT, and would need to be answered in a separate question (the answer could simply be a server setting for the maximum request size).
On a logical level, after you have the data in the Java Object. For this, I would recommend a validation/authorization layer. One of the awesome features of GWT is, that you can use JSR 303 validation both on the server and client side now. It doesn't cover every aspect (you would still have to test for user permissions), but it can cover your "#Size(max = 20)" use case.
I've been studying about how I could develop a distributed architecture that implements the protocol request/response using the concept of concurrency through actors.
I concluded that the best way to do this is by creating a response system with synchronous handling of Futures/Promises, and soon after the response, leaving an open channel to receive notifications.
Thus an architecture that would work exactly like a inbox message.
It has some problems.
Thus I would have to maintain two endpoints (actors in the two layers)?
The Problem:
The view module requests that a particular element is processed. She sends this command to be processed via RemoteActor on the application server. This server should immediately return the promise that it will notify you when the element is processed. After this, the view module will be waiting the notification of completion of processing.
How do you see this problem?
I am using Scala, Akka and Google Guice.
I believe it is a generic problem that everyone can make use of their solutions. Excuse me if I'm hurting the terms of stackoverflow site.
Thanks in advance
I don't want to distract from any good answers you may get on Akka, because I unfortunately don't know much about Akka and it's distributed actors features, but I'd like to ask if you've considered any alternatives.
It seems that you basically need an asynchronous RPC library. There are two powerful libraries written in Scala that I know of that may satisfy your requirements - http://sna-projects.com/norbert/ and http://twitter.github.com/finagle/. Finagle gives some really powerful combinators for expressing asynchronous computation dependencies and registering listeners on futures. I'm currently maintaining Norbert, which we use at LinkedIn for some of our distributed systems, such as search.
//On "server" node
class SomeActor extends Actor {
def receive = {
case messageToRemoteActor => self reply_? TypeOfResponseMessage()
}
}
Actor.remote.register("myRemoteActor", actorOf[SomeActor])
//On "client" node
val remoteActor = Actor.remote.actorFor("myRemoteActor", "hostnameOrIpOfNodeContainingTheActorAbove", port)
val f: Future[TypeOfResponseMessage] = remoteActor !!! messageToRemoteActor
f.onComplete( _.value.get match {
case Left(exception) => handle(exception)
case Right(result) => handle(result)
})
Why not just use one of 0MQ's REQ-REP sockets?
https://github.com/zcox/akka-zeromq-java
That way you solve your immediate problem and at the same time, begin learn an architecture that will take you a long way and supports communications with clients written in many languages.
For an example of where this might lead look at http://blog.getintheloop.eu/2009/05/22/lift-amqp-with-rabbitmq-and-scala-tutorial-and-screencast/
Note that I am NOT suggesting that you use AMQP today since a RabbitMQ broker would be overkill for your immediate problem. Rather, I am suggesting that you invest your time in using an architecture (message queueing) that will pay you dividends in the long run.
Recently, while reading a Socket Programming HOWTO the following section jumped out at me:
But if you plan to reuse your socket for further transfers, you need to realize that there is no "EOT" (End of Transfer) on a socket. I repeat: if a socket send or recv returns after handling 0 bytes, the connection has been broken. If the connection has not been broken, you may wait on a recv forever, because the socket will not tell you that there's nothing more to read (for now). Now if you think about that a bit, you'll come to realize a fundamental truth of sockets: messages must either be fixed length (yuck), or be delimited (shrug), or indicate how long they are (much better), or end by shutting down the connection. The choice is entirely yours, (but some ways are righter than others).
This section highlights 4 possibilities for how a socket "protocol" may be written to pass messages. My question is, what is the preferred method to use for real applications?
Is it generally best to include message size with each message (presumably in a header), as the article more or less asserts? Are there any situations where another method would be preferable?
The common protocols either specify length in the header, or are delimited (like HTTP, for instance).
Keep in mind that this also depends on whether you use TCP or UDP sockets. Since TCP sockets are reliable you can be sure that you get everything you shoved into them. With UDP the story is different and more complex.
These are indeed our choices with TCP. HTTP, for example, uses a mix of second, third, and forth option (double new-line ends request/response headers, which might contain the Content-Length header or indicate chunked encoding, or it might say Connection: close and not give you the content length but expect you to rely on reading EOF.)
I prefer the third option, i.e. self-describing messages, though fixed-length is plain easy when suitable.
If you're designing your own protocol then look at other people's work first; there might already be something similar out there that you could either use 'as is' or repurpose and adjust. For example; ISO-8583 for financial txns, HTTP or POP3 all do things differently but in ways that are proven to work... In fact it's worth looking at these things anyway as you'll learn a lot about how real world protocols are put together.
If you need to write your own protocol then, IMHO, prefer length prefixed messages where possible. They're easy and efficient to parse for the receiver but possibly harder to generate if it is costly to determine the length of the data before you begin sending it.
The decision should depend on the data you want to send (what it is, how is it gathered). If the data is fixed length, then fixed length packets will probably be the best. If data can be easily (no escaping needed) split into delimited entities then delimiting may be good. If you know the data size when you start sending the data piece, then len-prefixing may be even better. If the data sent is always single characters, or even single bits (e.g. "on"/"off") then anything different than fixed size one character messages will be too much.
Also think how the protocol may evolve. EOL-delimited strings are good as long as they do not contain EOL characters themselves. Fixed length may be good until the data may be extended with some optional parts, etc.
I do not know if there is a preferred option. In our real-world situation (client-server application), we use the option of sending the total message length as one of the first pieces of data. It is simple and works for both our TCP and UDP implementations. It makes the logic reasonably "simple" when reading data in both situations. With TCP, the amount of code is fairly small (by comparison). The UDP version is a bit (understatement) more complex but still relies on the size that is passed in the initial packet to know when all data has been sent.