Could you suggest a portable library for creating realtime bi-directional sockets that do not rely on the HTTP/S or any other plain text protocol? I'd like to send data up and down in the smallest chunks possible.
The protocol must support TLS
Everything to do with WebSockets is off the list.
Though portable, I'm specifically looking for something that will work in Python and Java (Android)
Thanks.
If you google RTP you might find some interesting resources. Among other things, this is what ip-telephony likes to use.
Related
Ok, let's say that there's a company (let's say e-commerce) that has to implement a support chat on its website. Do you know what are protocols/infrastructures needed?
I've seen XMPP but I don't know if it's really useful in my case...
Ty a lot!
If you need to implement all things yourself, and don't want to use XMPP server, you can create your own chat protocol.
Such a protocol can be based on JSON-RPC.
Also, you can make a simple protocol using only features from Socket.IO.
You can use Socket.IO both on the server and client-side.
You will also need a database for storing chats and messages.
If you anticipate very heavy traffic, you can use e.g Apache CASSANDRA, but a plain SQL (or noSQL) database will be sufficient for most implementations.
It is better to use a database you are familiar with. The PostgreSQL can be a good choice because it has nice JSON support.
If you prefer a noSQL database, MongoDB will be OK.
If you use React, for the frontend I recommend you to use a #chatscope/chat-ui-kit-react components together with the #chatscope/use-chat library (I'm the author of both of them).
First off, I intend no hostility nor neglegence, just want to know people's thoughts. I am looking into bi-directional communication between client and server; client being a web application. At this point I have a few options: MS-proprietary duplex binding, from what I hear unreliable and unnatural: comet, and web sockets (for supported browsers).
I know this question has been asked in other ways here, but I have a more specific question to the approach. Considering web sockets are client-side, the client code sits in JavaScript. Is it really the intention to build a large chunk of an application directly in JavaScript? Why didn't W3C do this in web services? Wouldn't it be easier if we were to be able to use SOAP to provide a contract and define events along with the existing messaging involved? Just feels like the short end of the stick so far.
Why not make it simple and take advantage of JS dynamic nature and leave the bulk of code where it belongs....on the server?
Instead of
mysocket.send("AFunction|withparameters|segmented");
we could say
myServerObject.AFunction("that", "makessense");
and instead of
...
mysocket.onmessage = function() { alert("yay! an ambiguous message"); }
...
we could say
...
myServerObject.MeaningfulEvent = function(realData) { alert("Since I have realistic data...."); alert("Hello " + realData.FullName); }
...
HTML 5 took forever to take hold....did we waste a large amount of effort in the wrong direction? Thoughts?
Sounds to me like you've not yet fully grasped the concepts around Websockets. For example you say:
Considering web sockets are client-side
This is not the case, sockets have 2 sides, you could think of these as a Server and a Client, however once the connection is established the distinction blurs - you could then also think of the client and the server as "peers" - each can write or read in to the pipe that connects them (the socket connection) at any time. I suspect you'd benefit from learning a little more about HTTP works on top of TCP - WebSockets is similar / analogous to HTTP in this way.
Regarding SOAP / WSDL, from the point of view of a conversation surrounding TCP / WebSocket / HTTP you can think of all SOAP / WSDL conversations as being identical to HTTP (i.e. normal web page traffic).
Finally, remember the stacked nature of network programming, for instance SOAP/WSDL looks like this:
SOAP/WSDL
--------- (sits atop)
HTTP
--------- (sits atop)
TCP
And WebSockets look like this
WebSocket
--------- (sits atop)
TCP
HTH.
JavaScript allows clients to communicate via HTTP with XMLHttpRequest. WebSockets extends this functionality to allow JavaScript to make arbitrary network I/O (not just HTTP), which is a logical extension and allows all sorts of applications that need to use TCP traffic (but might not be using the HTTP protocol) to be ported to JavaScript. I think it is rather logical that, as applications continue to move to the cloud, that HTML and JavaScript support everything that is available on the desktop.
While a server can do non-HTTP network I/O on behalf of a JavaScript client and make that communication available over HTTP, this is not always the most appropriate or efficient thing to do. For example, it would not make sense to add an additional round-trip cost when attempting to make an online SSH terminal. WebSockets makes it possible for JavaScript to talk directly to the SSH server.
As for the syntax, part of it is based on XMLHttpRequest. As has been pointed out in the other posting, WebSockets is a fairly low-level API that can be wrapped in a more understandable one. It is more important that WebSockets support all the necessary applications than that it have the most elegant syntax (sometimes focusing on the syntax can lead to more restrictive functionality). Library authors can always make this very general API more manageable to other application developers.
As you noted WebSockets has low overhead. The overhead is similar to normal TCP sockets: just two bytes more per frame compared to hundreds for AJAX/Comet.
Why low-level instead of some sort of built-in RPC functionality? Some thoughts:
It's not that hard to take an existing RPC protocol and layer it on a low-level socket protocol. You can't go the opposite direction and build a low-level connection if the RPC overhead is assumed.
WebSockets support is fairly trivial to add to multiple languages on the server side. The payload is just a UTF-8 string and pretty much every language has built-in efficient support for that. An RPC mechanism not so much. How do you handle data type conversions between Javascript and the target language? Do you need to add type hinting on the Javascript side? What about variable length arguments and/or argument lists? Do you build these mechanisms if the language doesn't have a good answer? Etc.
Which RPC mechanism would it be modeled after? Would you choose an existing one (SOAP, XML-RPC, JSON-RPC, Java RMI, AMF, RPyC, CORBA) or an entirely new one?
Once client support is fairly universal, then many services that have normal TCP socket will add WebSockets support (because it's fairly trivial to add). The same is not true if WebSockets was RPC based. Some existing services might add an RPC layer, but for the most part WebSockets services would be created from scratch.
For my noVNC project (VNC client using just Javascript, Canvas, WebSockets) the low-overhead nature of WebSockets is critical for achieving reasonable performance. Until VNC servers include WebSockets support, noVNC includes wsproxy which is a generic WebSockets to TCP socket proxy.
If you are thinking about implementing an interactive web application and you haven't decided on server-side language, then I suggest looking at Socket.IO which is a library for node (server-side Javascript using Google's V8 engine).
In addition to all the advantages of node (same language on both sides, very efficient, power libraries, etc), Socket.IO gives you several things:
Provides both client and server framework library for handling connections.
Detects the best transport supported by both client and server. Transports include (from best to worst): native WebSockets, WebSockets using flash emulation, various AJAX models.
Consistent interface no matter what transport is used.
Automatic encode/decode of Javascript datatypes.
It wouldn't be that hard to create a RPC mechanism on top of Socket.IO since both side are the same language with the same native types.
WebSocket makes Comet and all other HTTP push type techniques legible by allowing requests to originate from the server. It is kind of a sandboxed socket and gives us limited functionality.
However, the API is general enough for framework and library authors to improve on the interface in whichever way they desire. For example, you could write some RPC or RMI styled service on top of WebSockets that allows sending objects over the wire. Now internally they are being serialized in some unknown format, but the service user doesn't need to know and doesn't care.
So thinking from a spec authors POV, going from
mysocket.send("AFunction|withparameters|segmented");
to
myServerObject.AFunction("that", "makessense");
is comparatively easy and requires writing a small wrapper around WebSockets so that serialization and deserialization happens opaquely to the application. But going in the reverse direction means the spec authors need to make a much more complex API which makes for a weaker foundation for writing code on top of it.
I ran into the same problem where I needed to do something like call('AFunction', 'foo', 'bar') rather than serialize/de-serialize every interaction. My preference was also to leave the bulk of code on the server and just use the Javascript to handle the view. WebSockets were a better fit because of its natural support for bi-directional communication. To simplify my app development, I build a layer on top of WebSockets to make remote method calls (like RPC).
I have published the RMI/RPC library at http://sourceforge.net/projects/rmiwebsocket/. Once the communication is setup between the web-page and the servlet, you can execute calls in either direction. The server uses reflection to call the appropriate method in the server-side object and client uses Javascript's 'call' method to call the appropriate function in the client-side object. The library uses Jackson to take care of the serialization/deserialization of various Java types to/from JSON.
WebSocket JSR was negotiated by a number of parties (Oracle, Apache, Eclipse, etc) all with very different agendas. It's just as well they stopped at message transport level and left higher level constructs out. If what you need is a Java to JavaScript RMI, check out the FERMI Framework.
I was wondering if it is possible to have a RESTful web service and a bidirectional communication with the clients and server.
In my case the state on the server can change, so the server should send a message to the clients to update themself. Perhaps that's totally against the RESTful idea.
Can I AJAX help me with this issue?
Thanks!
Not really possible under the standard http paradigm, but check out Comet for a possible workaround on that problem and there is alway polling.
The functionality you are after is treated by the concept of web sockets, but they are not mainstream yet.
To keep your solution RESTful you can have the clients poll your service. You can optimize any number of ways, like implementing a special method that lets clients query for changes given a timestamp, then the client just keeps track of when it last checked.
You should take a look at BOSH. BOSH is similar to Comet, but more specific, and I think, there are more reliable implementations.
Though, you will have problems serving multiple users at the same time if you want to use a standard REST service. You should think of some other implementation using nonblocking IO.
There are probably more questions about bosh. Of course, there are websockets now too, but if you need to serve old Browsers, you cannot rely on them.
I've started my degree project, a mobile application suitable for iPhone, Android and (in the near future) Symbian. The server architecture is the following:
web site (for "standard" users);
web service (for mobile connections), based on TomCat and Axis2;
mySQL DB to storage users data.
Surfing across the web, I've read a lot of discussion about the interaction between the iPhone and Web Services, and I've to say that I've not a clear idea of what I can do and what not.
Let's start from the protocol used to retrieve data from the DB: the Android-side application uses SOAP protocol, can I do the same with iPhone? Are there some limitations or problems?
I have also read about the using of REST instead of SOAP, could it be possible with the server architecture described above? Which are the main advantages/disadvantages?
Sorry if these questions sound "n00b", but it's my first real experience with iPhone and the lot of informations found on the web messed up my mind and I'm scared to be confused. Forgive me for any error.
SOAP is simply too heavy for mobile communications. Why do all the work to wrap requests in an additional XML layer you'll have to parse? You send more data than you need to, and impose greater CPU burden on client and server.
Use REST. If you are doing a cross-platform project JSON makes a great payload container, otherwise plists work well for sending data from the server.
You can definitely do SOAP on the iPhone. Here is a nice tutorial on the subject. After all, SOAP is a HTTP based protocol and you have all the libraries you need to do HTTP on the iPhone.
Having said that, RESTful APIs are simpler than SOAP, so you might want to consider them. They're also HTTP based so you won't have any problems on doing that on iPhone. On the server side, if you use Java, you will have to use JAX-RS to implement that part.
Hope it helps.
Google Buffers
If your looking for a language and platform agnostic solution have a look at Google Buffers. You can easily serialise objects for transmission over the wire.
This question should get you started in Objective-C.
JSON
I have also used JSON within iPhone Apps with great success. Again, relatively language and platform agnostic but much simpler than Google Buffers.
SOAP with Fast Infoset is suited for small devices:
JAX-WS 2.0 and its reference implementation support both Fast Infoset and MTOM/XOP. This article includes information about Web Service Performance for Fast Infoset vs. MTOM/XOP:
http://www.devx.com/xml/Article/35385/1954
Fast Infoset is optimized for small
devices that have bandwidth
constraints, and is supported by many
vendors such as Microsoft .NET and
.NET CF, Sun GlassFish, BEA WebLogic,
IBM SDK for Java 6.0 and others.
http://en.wikipedia.org/wiki/Fast_Infoset
Have you ever seen a good reason to create a custom binary rest protocol instead of using the basic http rest implementation?
I am currently working on a service oriented architecture framework in .Net responsible for hosting and consuming services. I don't want to be based on an existing framework like Remoting or WCF, because I want total flexibility and control for performing custom optimization.
So here I am trying to find the best protocol for handling this SOA framework. I like the request/response stateless connection nature of REST and the uri for defining resources, but I dislike the text based nature of HTTP.
Here are my arguments for disliking HTTP, correct me if I'm wrong:
First an evidence, parsing text is less efficient than parsing binary.
I'd prefer a fixed length binary header containing the content length and a binary content.
Second, there is no notion of sequence number for http requests, so the only way of associating a response with its request is the socket connection used to send the request and receiving the response.
This means there can only be one pending request at a time for a specified socket, so if a service consumer want to send multiple requests in parallel to a service, it need to open multiple socket to the server.
A custom rest protocol could define a sequence number for requests, so request and response would be associated with the sequence number instead of the socket and there could be multiple requests sent in parallel on the same socket.
I think there is no way to do this standardly with HTTP, it could be done with a custom text based protocol, but why not make it binary based to gain performance.
To add a little more context, my SOA framework does not need to be accessible from a non .Net consumer, so I have no restriction about using the .Net binary formatter or another custom binary formatter.
So am I making any sense for wanting a custom binary rest protocol? If you think I'm wrong please tell me your arguments.
Thanks.
REST is an architectural style for constructing web services. Roy Fielding articulated it based on his experience in designing HTTP, but it transcends HTTP. You can deploy a RESTful service over ordinary email exchange for instance.
The REST representations of your resources can be anything you like, though Roy really stresses that people should try to use very carefully designed, standard representations. There's nothing wrong with binary. In fact image representations like JPEG and PNG are binary. Google's Protocol Buffers gives you ways of creating compact binary representations of structured data too.
So the short answer is that you can certainly be RESTful and use binary representations and a home-grown binary substitute for HTTP.
I would actually very strongly recommend you use HTTP for efficiency, though. If you use your own protocol then you lose all the leverage provided by the wonderful HTTP caching infrastructure that stands between your server and its clients. The full client load falls right on your server instead of getting spread out over the intermediate caches.
10/4/2010: In our HTTP-based REST APIs we now support Java Serialized Object and Kryo binary representations in addition to XML, JSON, and XHTML. The Kryo serialization library performs significantly better than the others without requiring any special protocol. Another alternative to cut down on bandwidth is to use HTTP compression along with a textual representation.
The data of your packets can be whatever you like; be it XML, Plaintext, JSON, or just a binary format. I see no reason to force any specific one of these on yourself (use whatever is most fitting).
Though, when saying 'binary format', most people hear a fixed format of field-lengths and other really annoying things. Generally, if you aren't desperate from a data-transfer point of view, I see no reason to go this way [though you may do a binary serialisation of .net objects so they can be re-instantiated [or look at the 'protocol buffers' lib for .net]].
Summary: Seems okay to me. Whatever floats your boat.
Have you ever seen a good reason to create a custom binary rest protocol instead of using the basic http rest implementation?
I'm not sure I understand your terms. A REST representation can be completely binary, and still transported over pure HTTP. There's no need to invent a new protocol just to transmit binary data. Just reduce your requirements to a well-documented media type (which you're free to invent).
Regarding binary resource representations, Fielding himself argues that binary is not only acceptable, but may be required in some situations.
Regardless of whether you go straight binary, Base64 mixed with text, or text-only, don't forget the hypertext constraint if you plan on calling what you do "REST".
So am I making any sense for wanting a custom binary rest protocol?
If you mean you'd like to create a custom, binary-only, hypertext-driven media type - that's perfectly reasonable.
If you're talking about inventing some custom extension to HTTP, I would avoid that unless absolutely necessary (and what you describe doesn't sound to me like it rises to that level).
I want total flexibility and control
Based in this statement then I would suggest one of two choices. Either straight TCP/IP sockets, or WCF. These options give you by far the most flexibility. WCF is a great solution if you want to be transport agnostic and you want to start with a clean state as far as an application protocol is concerned.
REST imposes a set of constraints that restrict how your distributed application behaves in order to gain certain beneficial characteristics. If you see your service end points as ways to invoke operations then I suggest that REST will not fit your needs well at all.
Somehow, I don't think whether you send binary or text payloads should be your biggest concern at this point.