I am trying to learn messaging system. I have found that many options like
RabbitMq
NServiceBus
MassTransit
Kafka
...
My questions is:
What is the difference between each one?
What is the use of each?(usage scenario)
The best way to think of NServiceBus and MassTransit is as middleware that sits on top of messaging infrastructure like RabbitMQ, Azure Service Bus, or Kafka.
For a comparison between RabbitMQ and Kafka, see this.
For a comparison between NServiceBus and MassTransit, see this.
Related
I'm currently planning the architecture for an application that reads from a Kafka topic and after some conversion puts data to RabbitMq.
I'm kind new for Kafka Streams and they look a good choice for my task. But the problem is that Kafka server is hosted at another vendor's place, so I can't even install Cafka Connector to RabbitMq Sink plugin.
Is it possible to write Kafka steam application that doesn't have any Sink points, but just processes input stream? I can just push to RabbitMQ in foreach operations, but I'm not sure will Stream even work without a sink point.
foreach is a Sink action, so to answer your question directly, no.
However, Kafka Streams should be limited to only Kafka Communication.
Kafka Connect can be installed and ran anywhere, if that is what you wanted to use... You can also use other Apache tools like Camel, Spark, NiFi, Flink, etc to write to RabbitMQ after consuming from Kafka, or write any application in a language of your choice. For example, the Spring Integration or Cloud Streams frameworks allows a single contract between many communication channels
I have an application which makes use of RabbitMQ messages - it sends messages. Other applications can react on these messages but they need to know which messages are available on the system and what they semantically mean.
My message queuing system is RabbitMQ and RabbitMQ as well as the applications are hosted and administered using Kubernetes.
I am aware of
https://kubemq.io/: That seems to be an alternative to RabbitMQ?
https://knative.dev/docs/eventing/event-registry/ also an alternative to RabbitMQ? but with a meta-layer approach to integrate existing event sources? The documentation is not clear for me.
Is there a general-purpose "MQ-interface service availabe, a solution, where I can register which messages are sent by an application, how the payload is technically and semantically set up, which serialization format is used and under what circumstances errors will be sent?
Can I do this in Kubernetes YAML-files?
RabbitMQ does not have this in any kind of generic fashion so you would have to write it yourself. Rabbit messages are just a series of bytes, there is no schema registry.
I am trying to use https://github.com/nestjs/cqrs Nest.JS module. Seems there is impossible to make CommandBus and QueryBus implementations external like using Kafka or RabbitMQ under the hood and share command handling between many microservices. Am I wrong?
Thanks.
Kafka support was added in 6.7.0
https://github.com/nestjs/nest/issues/2361
As documented, the command bus is observable, so you could consume the events and forward them into a Kafka producer or consumer, as you need to
from the official Nest.JS docs we can see that
We don't support streaming platforms with log based persistance, such
as Kafka or NATS streaming because they have been created to solve a
different range of issues.
However, you can see here how to create a microservice with NATS https://docs.nestjs.com/microservices/nats which theorically is not supported as seen above.
I would like to use Kafka with Nest. Is it / Will it be supported?
Thank you in advance!
Kafka message broker is an ecosystem that container whole codebase and other actor of your project.
All microservice pattern that mentions in Basics TCP, Redis InMemCacH, MQTT, NATS, gRPC can abstract in Nest.js core scaffold.
If you want use KAFKA or RABBITMQ may restructure your project and change it some applications that should separated and work together with message and message queue.
Since IIB v10.0.0.7 I can use KafkaConsumer node to receive messages that was published on a Kafka topic.
I need some client which will be able to recieve message from Kafka and put it in IBM MQ and get message from IBM MQ and publish in to Kafka topic. I alredy have IIB and IBM MQ. Kafka is messaging system of one of integration systems.
Can I somehow put received message from Kafka in IBM queue using KafkaConsumer node and MQOutput node ? Or get message from queue with MQOutput and publish it to Kafka topic with KafkaProducer node ?
Or it is not a good idea to mix this tecnologies in such a way and should look for some another workaround ?
Hi you could use Kafka Connect connectors.
https://www.confluent.io/product/connectors/
There are community connectors for MQ.
Alternatively, if you're using IBM MessageHub, i.e. Kafka-as-a-service in the IBM Cloud, there you can have an MQ-to-Kafka bridge ran as a service itself.
https://console.bluemix.net/docs/services/MessageHub/messagehub088.html#bridges
I hear this question every week...
The article “Apache Kafka vs. Enterprise Service Bus (ESB)—Friends, Enemies, or Frenemies? (https://www.confluent.io/blog/apache-kafka-vs-enterprise-service-bus-esb-friends-enemies-or-frenemies/)” discusses why Kafka is not competitive but complementary to integration and messaging solutions (including IBM MQ) and how to integrate both.
IIB can write to IBM MQ and one could IBM MQ source connector to write to kafka.
https://docs.confluent.io/kafka-connect-ibmmq-source/current/
Usage of kafka or IIB will be use to use dependent. Kafka is your messaging platform with persistence, ability connect to different sources and sinks and if needed enrich the messages on the fly in real/near realtime.