I have a usecase where I have to collect user events and store it in Kafka. Is it efficient to directly use Kafka clients in mobile apps and websites to produce messages directly to Kafka instead of a middle layer. Is Kafka designed to handle millions of concurrent connections?
Technically you can do it. The main reason people do not do it, especially for mobile apps, is that it would rather be difficult to maintain a long-term evolution of the product, control its security or even its scale.
In the past years, Kafka Clients API has evolved dramatically (for the best but, it's still a change). It also includes authorization and authentication mechanisms, but there's not much freedom on what you can do with those. Kafka is not built upon a standardized protocol and technology-agnostic specification like JMS that could be thought as a bit more flexible.
Also, between major versions, compatibility is not guaranteed, as for most existing technologies. It may happen that you need to keep multiple server versions for a long time just because some mobile clients are still outdated and there's coupling to a specific client version, that itself is coupled to a specific server version.
On the other side and for the same reason, it could also happen that you would need to keep handling older versions of the messages schemas for a long time, just to keep older clients happy.
That's when HTTP and, more specifically, the API gateway pattern, comes into place.
HTTP APIs are easier to throttle, perform rate limiting, applying custom security policies, custom authentication/authorization strategies, etc. And they are based on a standard protocol that's used all over the Internet.
There are also advantages of using HTTP when you plan to have some partner integrations using your backend platform. As they can do it easily without changing the technology stack.
By not exposing Kafka to the outside clients, you can change the underlying technology stack later without impacting the clients. To be honest, although Kafka is a brilliant piece of technology, it's difficult to compete with HTTP for internet communication. Kafka actually provides a REST Proxy, an HTTP based client that could be possibly thought to be used for this sort of things.
There is a chat project I plan to develop and currently stuck while deciding, which XMPP server and client to use.
Front-end will be implemented using Ionic 3+ framework (fixed requirement).
Main features:
private chat capability (1 on 1)
group chat capability (and ability to add users to existing chat)
online status visibility
offline message delivery (online user sends a message to offline user, and the offline user gets it after going online, etc)
full-time server-side chat history storage
I need to select:
a client library which is convenient to implement with Ionic 3+
a server which enables all the above requirements
Also, the chat messages should cross the network with at least some kind of encryption.
Currently, I have been playing with these server options:
eJabberd - most people recommend this option, but for me, a person, who is newbie at XMPP stuff, it looks like there will be quite a lot of hassle with configuration
openFire - easy to setup and use, but it uses Java VM, and some people argue, that it has problems with performance and memory management.
I also stumbled upon AstraChat - a commercial product, that looks similar to what I need - however, with fixed implementation of it's own public apps - no way to use it's code base.
As of front-end - I have only tried running the Ionic 1 tutorial locally using Strophe.js as the client in the front-end. Since I am developing the app with Ionic 3+ (TypeSript), it would be awesome to use some newer and TS compatible library - however, it is just a wish, not a requirement. :)
Maybe there are some better and more convenient non-XMPP implementations for this concept. Any thoughts are welcome!
eJabbered is very complex decision for your application.
openFire is also very complex and requires JVM
For your apllication I advise Prosody. It is simple and lightweight desicion.
https://en.wikipedia.org/wiki/Comparison_of_XMPP_server_software
But please think twice about using chat on XMPP protocol. This protocol is very difficult to learn and it has a variety of extensions(e. g. XEP) and you will be confused about documentation.
Alternative https://github.com/actorapp/actor-platform
but documentation is obsolete.
We have a OTT application which deliver message to users over XMPP protocol. we using eJabberd to deliver messages. also we have group chats that user can use them with maximum of 200 users.
Now we want to build channels. some think like Telegram channels.
it seems similar application like Viber and Telegram using different infrastructure for channels.
Now the question is what is the best infrastructure for channels (server side and client side)?
where is our start point?
There is many approach you can use to build such system. A tool like ejabberd has been designed to build that type of system. It is scalable and fully extensible through a built in API. This would be a good bet as many OTT services are already built on top of ejabberd.
Looking for ideas on how to implement a kiosk style/full screen client that will display messages sent to it, in realtime or push based. Basically, think a 911 dispatch center call board. I was thinking a xmpp bot to display, and making a simple xmpp client that can only send messages to the bot.
This is for a very small emergency disaster agencies war room, and only needs to be able to display simple messages entered in from one of the computers in the building. Is XMPP a good solution for this?
An IM protocol like XMPP is an acceptable solution for this.
XMPP seems to fit the bill, it is mature and has many clients that support it so it would not be necessary to write a client, just set up a regular IM client to send to the "buddy" that is the big board.
Are you looking at the problem backwards? Describe what you want to do a little more THEN seek advice on protocols. It smells like you might be designing around a protocol rather than designing around your requirements.
This should be as simple as a single HTML page, running full screen, using Strophe.js and an XMPP account. Strophe is an easy-to-use XMPP library in Javascript.
Something like the basic.{html,js} example here should be pretty much what you want:
http://code.stanziq.com/cgit/strophe/strophejs/tree/
Sounds like a simple pubsub setup (XMPP will work for this) where the clients are all publishers and the War room is the only subscriber. This eliminates the need for rosters so it keeps the intial configuration pretty simple.
I don't know what language you prefer to use, but it would be rather simple in Smack using the pubsub API and any XMPP server you prefer that supports the pubsub extension. (You will need to build Smack from source though as that particular API is new and not in the release version yet.)
I have heard a lot of people talking recently about middleware, but what is the exact definition of middleware? When I look into middleware, I find a lot of information and some definitions, but while reading these information and definitions, it seems that mostly all 'wares' are in the middle of something. So, are all things middleware?
Or do you have an example of a ware that isn't middleware?
Lets say your company makes 4 different products, your client has another 3 different products from another 3 different companies.
Someday the client thought, why don't we integrate all our systems into one huge system. Ten minutes later their IT department said that will take 2 years.
You (the wise developer) said, why don't we just integrate all the different systems and make them work together? The client manager staring at you... You continued, we will use a Middleware, we will study the Inputs/Outputs of all different systems, the resources they use and then choose an appropriate Middleware framework.
Still explaining to the non tech manager
With Middleware framework in the middle, the first system will produce X stuff, the system Y and Z would consume those outputs and so on.
Middleware is a terribly nebulous term. What is "middleware" in one case won't be in another. In general, you can expect something classed as middleware to have the following characteristics:
Primarily (usually exclusively) software; usually doesn't need any specialized hardware.
If it weren't there, applications that depend on it would have to incorporate it as part of their application and would experience a lot of duplication.
Almost certainly connects two applications and passes data between them.
You'll notice that this is pretty much the same definition as an operating system. So, for instance, a TCP/IP stack or caching could be considered middleware. But your OS could provide the same features, too. Indeed, middleware can be thought of like a special extension to an operating system, specific to a set of applications that depend on it. It just provides a higher-level service.
Some examples of middleware:
distributed cache
message queue
transaction monitor
packet rewriter
automated backup system
Wikipedia has a quite good explanation: http://en.wikipedia.org/wiki/Middleware
It starts with
Middleware is computer software that connects software components or applications. The software consists of a set of services that allows multiple processes running on one or more machines to interact.
What is Middleware gives a few examples.
There are (at least) three different definitions I'm aware of
in business computing, middleware is messaging and integration software between applications and services
in gaming, middleware is pretty well anything that is provided by a third-party
in (some) embedded software systems, middleware provides services that applications use, which are composed out of the functions provided by the hardware abstraction layer - it sits between the application layer and the hardware abstraction layer.
Simply put Middleware is a software component which provides services to integrate disparate systems together.
In an complex enterprise environment, there are a number of challenges when you need to integrate two or more enterprise systems together to talk to each other. Normally these systems do not understand each others language as they are developed on different platforms using different languages (like C++, Java, Cobol, etc.).
So here comes middleware software in picture which provides services like
transformation of messages formats from one app to other,
routing and enriching messages besides taking care of security,
encryption,
validation and
applying different business rules to these messages.
A typical example of middleware is an ESB products like IBM message broker (WMB/IIB), WESB, Datapower XI50, Oracle Fusion, Mule and many others.
Therefore, middleware sits mostly in between the service consuming apps and services provider apps and help these apps to talk to each other.
Middleware is about how our application responds to incoming requests. Middlewares look into the incoming request, and make decisions based on this request. We can build entire applications only using middlewares. For e.g. ASP.NET is a web framework comprising of following chief HTTP middleware components.
Exception/error handling
Static file server
Authentication
MVC
As shown in the above diagram, there are various middleware components in ASP.NET which receive the incoming request, and redirect it to a C# class (in this case a controller class).
Middleware is a general term for software that serves to "glue together" separate, often complex and already existing, programs. Some software components that are frequently connected with middleware include enterprise applications and Web services.
There is a common definition in web application development which is (and I'm making this wording up but it seems to fit): A component which is designed to modify an HTTP request and/or response but does not (usually) serve the response in its entirety, designed to be chained together to form a pipeline of behavioral changes during request processing.
Examples of tasks that are commonly implemented by middleware:
Gzip response compression
HTTP authentication
Request logging
The key point here is that none of these is fully responsible for responding to the client. Instead each changes the behavior in some way as part of the pipeline, leaving the actual response to come from something later in the sequence (pipeline).
Usually, the middlewares are run before some sort of "router", which examines the request (often the path) and calls the appropriate code to generate the response.
Personally, I hate the term "middleware" for its genericity but it is in common use.
Here is an additional explanation specifically applicable to Ruby on Rails.
Middleware stands between web applications and web services that natively can't communicate and often are written in different languages/frameworks.
One such example is OWIN middleware for .NET environment, before owin people were forced to host web apps in a microsoft hosting software called IIS. After owin was developed, it has added capacity to host both in IIS and self host, in IIS was just added support for Owin which acted as an interface. Also it become possible to host .NET web apps on Linux via Mono, which again added support for Owin.
It also added capacity to create Single Page Applications, Owin handling Http request/response context, so on top of owin you can add authentication/authorization logic via OAuth2 for example, you can configure middleware to register a class which contains logic of user authentification (for ex. OAuth2 implementation) or class which contains logic of how to manage http request/response messages, that way you can make one application communicate with other applications/services via different data format (like json, xml, etc if you are targeting web).
Some examples of middleware: CORBA, Remote Method Invocation (RMI),...
The examples mentioned above are all pieces of software allowing you to take care of communication between different processes (either running on the same machine or distributed over e.g. the internet).
From my own experience with webwork, a middleware was stuff between users (the web browser) and the backend database. It was the software that took stuff that users put in (example: orders for iPads, did some magical business logic, i.e. check if there are enough iPads available to fill the order) and updated the backend database to reflect those changes.
It is just a piece of software or a tool on which your application executes and rapplication capabilities with respect to high availability,scalability,integrating with other softwares or systems without you bothering about your application level code changes .
For example : The operating system on which your application runs requires an I.P change , you do not have to worry about it in your code , it is the middleware stack on which you can simple update the configuration.
Example 2 : You experience problems with your runtime memory allocation and feel that the your application usage has increased , you do not have to much about it unless you have a bug or bottleneck in your code , it is easily achievable by tuning middleware software configuration on which your application runs.
Example 3 : You have multiple disparate software and you need them to talk to each other or send data in a common format which is understandable by all the systems then this is where middleware systems comes handy.
Hope the information provided helps.
it is a software layer between the operating system
and applications on each side of a distributed computing system in a network. In fact it connects heterogeneous network and software systems.
If I am not wrong, in software application framework, based on the context, you can consider middleware for the following roles that can be combined in order to perform certain activities in between the user request and the application response.
Adapter
Sanitizer
Validator
I always thought of it as the oldest software I have had to install. The total app used a web server, a database server, and an application server. The web server being the middleware between the data and the app.