I've been looking into Apache Camel and Kafka over the past two days in hopes of learning about messaging frameworks/brokers. Is a possible use case of Camel/Kafka using Kafka as the message broker while implementing the producers and consumers with Camel? I saw a brief example of something similar, but can't seem to find it again. If not, what is the point of the Camel:Kafka component?
Yes Apache Camel makes using Kafka easier as it hides a bunch of the complexities, which is the main point about Camel components. However if you need to do something really advanced or be in control yourself then sometimes a Camel component may lack a functionality for that, or some hooks/apis you need, and if so people ask for that and we improve these components over time, to the communities requirements. And if you cannot wait/do that, then you do NOT have to use a Camel component and can use the Kafka API yourself directly - after all its all just Java.
There is also an Camel example here: https://github.com/apache/camel/tree/master/examples/camel-example-kafka.
And the Camel in Action 2nd edition book covers Camel with Kafka in its cluster chapter.
Related
I am considering different message-oriented middleware which I may use on our information system. I've been reading lots of papers and framework documentation. I see that Kafka is not the first choice when we want to do SAGA.
As an example, I read MassTransit's riders documentation, and it says that "many of concepts and idioms" that are used on classical transports (e.g. ActiveMQ, RabbitMQ, etc.) do not apply on Kafka.
What is not possible with Kafka that is usually possible with other MQs (e.g. ActiveMQ, RabbitMQ, etc.)?
What, from a transactional point of view, makes it more complicated to use Kafka in a Saga implementation?
More than a question, this is an architectural dilemma that I am facing.
Is it a good idea to have REST wrapper around a Kafka Producer and integrate with it, instead of directly integrating with Kafka Producer in my code? I could use a generic interface for my higher classes, instead of using the KafkaImpl directly to keep it loosely coupled for the future.
If you have the option, I'd probably go for the pure-Kafka approach, since you'd get better throughput (the clients are very intelligent with respect to batching and futures).
I'm not sure you're decoupling your code by adding a rest wrapper; you're just adding another level of abstraction, adding maintenance burden and covering over some of the benefits of Kafka.
If you really need to use REST, you can make use of Kafka-Rest - no need to reinvent the wheel!
As mentioned kafka-rest-proxy will work.
I know plenty of people that wrap Kafka producer/consumers with Spring Kafka, Mirconaut, Akka, Quarkus, Lagom/Play just to name a few. Spring, specifically, has the messaging binders that can provide that "generic interface" feel.
These are all web frameworks, and putting an API / RPC abstraction layer on any code is definitely necessary in 12factor applications
I have just started learning Kafka and I am trying to build a prototype to have a producer which is a REST API and send the data to Kafka consumer. I went through quite a lot of documentation to figure out some particular procedure.
I wasn't able to understand if there is a single connector that I could just use just like the fileconnector or the JDBC connectors provided for Apache Kafka. Should I be writing a custom connector for this ?
I am pretty much confused on where to start. I am particularly looking for some structured documentation or idea on how to get this done.
It sounds like you're talking about the functionality that already exists in the REST Proxy. This provides a REST API for producing data into Kafka, or consuming data from Kafka.
Edit: From your comment I understand your question to be different. If you want to pull data from a REST endpoint into Kafka you can use Kafka Connect and the kafka-connect-rest plugin. There's an example of it in use here.
There's no need to write a connector (besides, it's not possible with python). One already exists for HTTP.
https://github.com/llofberg/kafka-connect-rest
For a Java/Kotlin Spring boot app, if I want to send messages to Kafka or consume messages from Kafka. Would you recommend using Spring Kafka library or just using Kafka Java API.
Not quite sure are there any more benefits Spring provides or just a wrapper? For Spring they provide a lot of annotations which seems more magics when having some runtime error.
Want to hear some opinions.
Full disclosure: I am the project lead for Spring for Apache Kafka.
It's entirely up to you and your colleagues.
It's somewhat comparable to writing assembly code Vs. using a high level language and a compiler.
For an existing Spring shop that is familiar with spring-messaging (JMS, RabbitMQ etc), it's a natural fit, the APIs will be very familiar (POJO listeners, MessageConverters, KafkaTemplate, etc, etc).
When using the simplest APIs, Spring takes care of the low-level stuff like committing offsets, transaction synchronization, error handling, etc, etc.
If you have very basic requirements and/or want to write all that code yourself, then use the native APIs.
I am trying to understand the differences between something like Kafka and something like Camel. To my understanding Camel would provide much more abstraction for developers without having to worry about changing protocols/systems to some extent. How would Kafka not be able to handle most of what Camel can do now? I am reading through the documentation and it seems like Kafka has been updated/upgraded enough to slightly break away from being a message broker only. I guess my question would really come down to how does Kafka compare to Camel in regards to future proofing systems and where does Kafka fall short of Camel? I am under the impression that Kafka doesn't scale as well as a system grows.
Edit: This is strictly based around messages.The documentation surrounding Camel makes it very clear that it's based around Enterprise Integration Patterns, but the deeper I dive into Kafka documentation the same patterns can be implemented. Am I missing something?
Apache Kafka : Is a streaming processing platform. It is based on massively scalable publish subscribe message queue architecture. There are many other platforms which are based on JMS publish subscribe model, which could do the same(with some exceptions). Some of the most popular are Apache-Activemq, RabbitMq
Apache Camel : Is a message oriented middleware. It has implemented almost all the Enterprise Integration Patterns.
You can use Apache Camel with Apache Kafka. Or you can use Apache Kafka without Apache Camel also.
They are two totally different things.
Think about Camel as an interface definition tool where you can define endpoints or channels where messages fly in. But they are abstract. Compare Camel with Spring Integration for instance.
Kafka can provide those messages, so it can implement those abstract channels or endpoints. But so can ActiveMQ and others.
Kafka is a message broker. It is comparable with other message brokers like ActiveMQ, RabbitMQ, Azure Service Bus etc. Camel is an integration middleware. It is more comparable to Apache ServiceMix.
Taking a look at the theory of an Event-Driven Architecture https://www.oreilly.com/library/view/software-architecture-patterns/9781491971437/ch02.html we could differentiate two different kinds of Event-driven topologies depending on whether we need an event mediator or not.
Message broker. In this category we find Kafka as it doesn't rely on a message mediator. Of course as written on previous answers, we could use Kafka together with a mediator depending on our needs.
Message mediator. In this category we find products like Camel. You may see it as a message controller.