I want suggestions of alternatives to Spring-Kafka.
I have used Kafka integrated in Spring in my application. I want to explore any alternative libraries available. Would be good if comparative analysis is share between libraries.
In addition to the plain Java API ... Put site:github.com + kafka in your favorite search engine...
wix/greyhound (seems pretty simple, and based on ZIO)
zio-kafka (ZIO is cool)
ExpediaGroup/rhapsody (Worth a look)
dropwizard/dropwizard-kafka (my personal 2nd choice)
Reactor Kafka (I think Spring-Kafka / Spring Cloud Streams kinda took over this project)
Akka Alpakka (Akka is a great framework)
SmallRye Reactive Messaging (with or without Quarkus) (one I would pick for new projects)
Micronaut Kafka (Good for webapps)
Vert.x Kafka client (Reactive, functional programming is awesome)
... probably more
Related
when consuming kafka messages, what are the advantages of flink over a simple golang/java program using for loops to consume kafka messages?
thanks in advance~
I understand that flink has better support for exactly once assurance, but are there any other advantages?
Forget processing guarantees. Flink is an ETL tool. It requires a scheduler to actually deploy any job, and has builtin libraries for SQL and connectors to external systems.
If all you care about is a lightweight binary for a Kafka-only process, then absolutely use Go, Rust, whatever. But you still need to bring-your-own scheduler (Kubernetes, perhaps?). That app won't run in Flink/YARN on its own...
BTW, Apache Beam has a Go and Java SDK, and a Flink runtime layer, so best of both?
My project has a lot of Spring Batch Jobs.
I have requirement to create externalized configuration for message brokers (example Kafka, rabbitMQ etc.).
I want to use spring cloud stream since it has various binders to solve this problem.
Hence i wanted to understand if there is an integration between the two frameworks, please explain.
Whats the difference between Spring Cloud Kafka Streams Vs Spring Cloud Stream Vs Spring Cloud Function Vs Spring AMQP and Spring for Apache Kafka?
Spring for Apache Kafka and Spring AMQP are foundational libraries for writing Spring friendly applications for Apache Kafka and AMQP respectively. They provide design patterns such as templates, message listener containers, and a wide array of other mechanisms to interact with the middleware systems at a lower level. These libraries do not require Spring Boot, but Spring Framework is the least common denominator. In other words, you can write a traditional Spring application with only Spring Framework contexts using these libraries.
Spring Cloud Function is a library that is part of the Spring Cloud portfolio projects. This is used as part of Spring Boot applications. It gives a consistent programming model for writing applications that involve various paradigms such as request-response (HTTP), event-driven (pub-sub), stream-processing (pub-sub/streaming), reactive streams, etc. The programming model at the application level is through the Java 8 functional model - for example you can write your business logic as a java.util.function.Function<?, ?>. Spring Cloud Function is not coupled with any middleware or other such technologies.
Spring Cloud Stream is another Spring Cloud project that is specifically built for event-driven and stream-processing usecases. Because this is a Spring Cloud project, it requires to be used as part of a Spring Boot application. The recent versions of Spring Cloud Stream is built on the foundations that Spring Cloud Function provides. This is essentially a destination binding framework that allows you to provide a destination - such as a Kafka topic or a RabbitMQ exchange. Spring Cloud Stream will bind those destinations for the application. The core Spring Cloud Stream does not have any middleware dependencies. That's where the binder implementations come in.
Spring Cloud Stream provides two kinds of Kafka binders - spring-cloud-stream-binder-kafka and spring-cloud-stream-binder-kafka-streams. The first one is a binder implementation where it provides programming model support for writing regular Kafka producers and consumers. For the most part, you can take this same application and provide another binder (such as spring-cloud-stream-binder-rabbit) and it should work (provided that the application makes the right configuration changes). This is because the binders are the ones concerned with lower-level details of communicating to the middleware and not the app itself. Apps can largely focus on the business logic at hand. The Kafka Streams binder in Spring Cloud Stream is a binder implementation specifically built for writing streaming applications using Kafka Streams. Both Kafka binder implementations use Spring for Apache Kafka under the hood.
The rabbit binder in Spring Cloud Stream uses Spring AMQP internally.
To summarize:
Spring for Apache Kafka/Spring AMQP - lower-level foundational libraries, do not require Spring Boot.
Spring Cloud Function - Spring Cloud project providing Java 8 functional programming model, Used with Spring Boot applicaitons.
Spring Cloud Stream - Framework for event-driven applications using Spring Cloud Function. Used with Spring Boot applications.
Spring Cloud Stream Kafka/Kafka Streams - Spring Cloud Stream binder implementation using Spring for Apache Kafka. Used with Spring Boot applications.
I'm new to Camel and Kafka. I have been reading some documentation on Camel, and have come across this repository with links to number of connectors. These seem to be connectors to primary sources of storage. What if I need to connect from .NET, pull the data and process it before committing it to my database? I feel like I'm missing the point somehow as I don't see any kinds of C# connectors.
Camel kafka connector project aims to provide a set of kafka connectors ready to use by simply configuring them. These connectors are based on the corresponding Camel components, that in turn focus on talking with an "external" system, protocol or SaaS.
So if I understood you usecase you probably need a kafka client in C# (like https://github.com/confluentinc/confluent-kafka-dotnet) so in your application logic you can connect to kafka grab events, apply what logic is needed and place them in the application database.
Basic question, which platforms and languages does Apache Kafka currently support?
Kafka is written in Scala, which means it runs on the JVM, so you can effectively run on any OS that supports the JVM. However, the brokers extract a huge performance boost by using the OS s kernel buffer cache. Im not sure how good this is with a non-unix system like Windows. The kafka source code base provides first class support for Scala and Java Clients . You could also find producer and consumer clients in languages like Php,C++, python etc under the contrib directory.
Apache Kafka runs well and is most stable and performant on Linux (either bare metal Linux, Linux VMs in private or public clouds, or Linux based docker containers). Kafka has been known to run on Windows but most vendors that commercially support Kafka do not extend their support to Windows for production servers so it's "community supported" by the Kafka community. Kafka also runs quite well on macOS for development.
The Apache Kafka distribution includes support for Java and Scala clients only but the larger Kafka community has created a long list of clients for other languages. A good list of the available options for clients is on the apache kafka wiki here: https://cwiki.apache.org/confluence/display/KAFKA/Clients
You will find that for some languages (like C#/.Net, Python, or Go) there are 2 or 3 or even more options for client libraries. Some are up to date with the newest Kafka wire protocol changes such as Exactly-Once Semantics, and message Headers which were added in Apache Kafka 0.11 or timestamps which were added in 0.10, or the security enhancements and new consumer api added in 0.9, and others are not. Some have the full set of functions/methods provided in Java (like seek(), or consumer group management, or interceptors) but others do not. Some are written purely in the target language and others are wrappers in the librdkafka C/C++ library. Some are commercially supported by a vendor and others are not, so choose based on your needs in terms of functionality, stability, execution environment, and supportability.