Sorry for my ignorance , but this subject has been stressing me for days
I currently read in a book about networks that there is a UseNet that is like a blog , but uses a different protocol to communicate
My question is, when I'm developing a software, can I decide which application protocol will it work?
the TOR browser, I read that .onion sites use a different protocol than HTTP, how is this possible?
We first need to break up your question. There are protocols implemented in operating systems (low-level protos) and those implemented in applications (high-level protos).
What makes early high-level protos so interesting, like Usenet and SNMP, is that they are not dependent on low-level protocols. Basically they just need the operating system to provide a mechanism for streaming data to come to/from the machine they want to speak to. You can see that Usenet's protocol's , UUCP, code can handle TCP or serial connections.
My question is, when I'm developing a software, can I decide which application protocol will it work?
Yes, you ask the OS for your low-level protocol and you then implement your high level protocol in you application. You can also layer protocols ontop of one another.
the TOR browser, I read that .onion sites use a different protocol than HTTP, how is this possible?
While not a TOR expert, general proxy protocols like SOCKS, use TCP at a low level to connect to their first proxy and then using higher level protocols encrypt HTTP requests so they bounce around several proxies.
this is a very high level answer, you can absolutely implement a low-level protocol in an application. I am just speaking to the most common examples
Related
Recently, I created a lightweight wrapper for the C++ boost asio library for some network communication. I used it to prototype some new functionality. We quickly moved over to a system that utilized Kafka to take advantage of an existing microservice framework when more internal funding came our way. No problem, I figured we would move to a different network model later on, and the internals were more important to my job than the network communication.
My question is, with the amount of technologies that abstract away network interfaces, (i.e. Kafka, grpc, ActiveMQ, ZeroMQ, etc.) are the use of base TCP/UDP sockets becoming more of a last resort, where software architects try to find an existing broker/stream processor/network message passing tool to fit their model? Or are there still many new production developments utilizing base level TCP/UDP sockets, not including those who solely write network libraries such as those mentioned above?
Note that I don't work with Kafka, grpc etc. in my line of work, but I have used UDP/TCP sockets extensively in the past. So forgive any misunderstanding of those particular technologies.
I want to dive into the whole diversity of tools which provide connection between programs over the network.
To clarify the question, I divide it on subquestions:
Why some groups of programs (or specific tools/frameworks/approaches with programming languages where this frameworks can be used) were popular in each period of time? (I expect description of problems which were solved, description of tools, why those tools are considered as best solution to those problems at that time, why some tools lost popularity)
What is the entire history of software communication over the network? (tools/approaches popularity precisely to decades)
What are the modern solutions to this problem?
I can distinguish only two significant approaches.
RPC, RMI and their implementations (I saw this, but it is about concrete problem and specific tools to solve this problem, I want to see the place of this problem in the whole picture of interconnection programs over the network. I heard about implementations: ONC RPC, XML-RPC, CORBA, DCOM, gRPC, but which are active now? which are reasonable to use? which are preferable and why? I want answers not to be opinion based, so I accept answers like "technology A better than technology B for problem X because ..." only if there is reliable research/statistics or facts). I heard that RPC and RMI were popular 10 years ago. Are they still?
Web services: REST, SOAP.
Am I miss something? Maybe there are some technologies which solve problem completely new way? Maybe there are technologies which can be treated as replacement to RPC(RMI) and Web Services? Can we replace RPC(RMI) by REST for any task? Can we replace RPC(RMI) by REST only for modern tasks? Should I separate technologies not as RPC and Web Services, but in some other manner?
As a partial answer, I can give you my feedback on the use of RabbitMQ.
As explain here, it provides a lot of different ways to use it :
RPC by implementing a "callback" queue
One to one, one to many routing strategy to propagate your events through your whole infrastructure and target the right destination.
It comes with the ability to persist messages to avoid loosing data when a crash appears but also with some plugins to increase possibilities (e.g x-delayed plugin)
This technologie written in Erlang is powerful and is a must try in term of communication between programs.
To your question „Am I missing something“: yes.
Very popular communication patterns are the so-called Event-Driven or Message-Driven protocols. This type of protocols are often used in distributed systems such web applications, microservices and IoT-Environments. The communication is complete asynchronously and allows building scalable and loosely coupled systems.
There are many different frameworks and methods for Event-Driven systems like WebSockets, WebHooks, Pub-Sub and Messaging-Librarys like AcitveMQ, OpenMQ, RabbitMQ, ZeroMQ and MQTT.
Hope this info helps for your research.
I will be developing a protocol that will encompass a client and a server module.
Several protocols that are close to the one I have in mind exist, but I want to make something simpler with less overhead and with something I have more control on.
The protocol I will be doing can and will be run almost in any scenario, local, web, lan, internet, etc. It can also run on a single box.
My question is, how can I start developing for the server side of the protocol?
Any ideas, insights, key words, starting points would be appreciated.
Regards
Do you want to develop the protocol yourself and then use it? or you are looking for a good protocol to use?
If second, I have recently worked with "Websocket" protocol which enables clients to communicate with each other via a server. It is a really good protocol and there are libraries in .NET ( >= 4.5), JAVA, Javascript, ... It is also supported in many browsers.
I am currently doing my research on building a chat system with more than 10k users connected online. I came across technologies and ways to do it such as jabber(XMPP), websockets, long polling, push. As far as I now, long polling might not work given the number of users. I know there is a lot of ways to accomplish this. I also know that facebook and Google chat systems are developed on XMPP.
I would truly appreciate if anyone could point me to the right direction. I believe all these methods and technologies out there are good depending on the scale of the project. I definitely need performance and scalability.
I've used Socket.io together with NodeJS for such a chat application. It scaled to over 10K concurrent users on moderate servers and there was a lot of room to grow.
This does depend on your limitations, tho.
What kind of hardware are you planning on using?
Which operating system would power your servers?
Which client platforms are you targeting?
Do you have an existing infrastructure you need to fit this into?
Do you have a previously selected programming language?
The existing skill sets your team members have and your team's ability to adopt new platforms and languages if necessary.
Take all of the above into consideration when making your decision.
Personally, I've found XMPP to be quite adequate, but a bit bloated for my purposes. YMMV.
You are comparing a fruit basket and three different variety of oranges.
XMPP is the only protocol that you have mentioned that actually is designed to support a chat system (of which many exist). The others are simply asynchronous messaging protocols/techniques. XMPP already supports http based chat via BOSH. Without a doubt, it will also support WebSockets when the specification is finalized. There is actually a draft of this already written, but at this point it appears to be a draft using a draft, so there will probably be few, if any, implementations.
Using XMPP would allow you to build on a proven technology for implementing a chat system and would allow you to choose what transport you want to use "under the hood". You haven't actually said whether you need a http based transport or not, but with XMPP you can use the stock tcp socket based transport or a http based one (BOSH) with the knowledge that it will also support WebSockets in the future.
The other benefit is of course that this is a widely used standard that will allow reuse of existing clients, servers and libraries in pretty much all popular (and not so popular) languages and platforms.
Scalability is not too much of a concern with the numbers you are quoting, as most (maybe all) existing xmpp servers will handle that many users.
At the moment our infrastructure doesn't support XMPP but this is what the big boys (Facebook, Google) seem to use as their protocol of choice. Is there anything that makes XMPP particularly suited to this application compared AMQP? In the end I believe in using the right tool for the job. If AMQP has any glaring weaknesses in this regard it is something I'm interested in finding about.
For one, AMQP does can not work over HTTP, or web sockets. So AMQP could be used inside your network, but not directly in your client browsers.
Instant messaging is the basic usage for XMPP and it is designed with this purpose in mind. It can certainly be used beyond that scope, but it is very good at this function.
Microsoft Messenger and Skype also offer XMPP for messaging.
AMQP is a more general purpose messaging protocol, and lacks things like presence which are fairly core to any IM protocol.