Why do not we have direct integration for Reactive Kafka in Spring Cloud Stream?
You can track this on Spring Kafka project on initial POC and I believe Spring Cloud Stream would make use of this once the final integration is done.
Related
I am migrating from Streamset to Spring Cloud Data Flow. When I am looking for module list I realized that some of the sources are not listed in Spring Cloud Flow - One of them is KAFKA source.
My question is why external KAFKA source is removed from standard sources list in spring cloud data flow ?
It is not that it is removed, but rather does not exist yet. See https://github.com/spring-cloud/stream-applications/issues/265
My project has a lot of Spring Batch Jobs.
I have requirement to create externalized configuration for message brokers (example Kafka, rabbitMQ etc.).
I want to use spring cloud stream since it has various binders to solve this problem.
Hence i wanted to understand if there is an integration between the two frameworks, please explain.
Whats the difference between Spring Cloud Kafka Streams Vs Spring Cloud Stream Vs Spring Cloud Function Vs Spring AMQP and Spring for Apache Kafka?
Spring for Apache Kafka and Spring AMQP are foundational libraries for writing Spring friendly applications for Apache Kafka and AMQP respectively. They provide design patterns such as templates, message listener containers, and a wide array of other mechanisms to interact with the middleware systems at a lower level. These libraries do not require Spring Boot, but Spring Framework is the least common denominator. In other words, you can write a traditional Spring application with only Spring Framework contexts using these libraries.
Spring Cloud Function is a library that is part of the Spring Cloud portfolio projects. This is used as part of Spring Boot applications. It gives a consistent programming model for writing applications that involve various paradigms such as request-response (HTTP), event-driven (pub-sub), stream-processing (pub-sub/streaming), reactive streams, etc. The programming model at the application level is through the Java 8 functional model - for example you can write your business logic as a java.util.function.Function<?, ?>. Spring Cloud Function is not coupled with any middleware or other such technologies.
Spring Cloud Stream is another Spring Cloud project that is specifically built for event-driven and stream-processing usecases. Because this is a Spring Cloud project, it requires to be used as part of a Spring Boot application. The recent versions of Spring Cloud Stream is built on the foundations that Spring Cloud Function provides. This is essentially a destination binding framework that allows you to provide a destination - such as a Kafka topic or a RabbitMQ exchange. Spring Cloud Stream will bind those destinations for the application. The core Spring Cloud Stream does not have any middleware dependencies. That's where the binder implementations come in.
Spring Cloud Stream provides two kinds of Kafka binders - spring-cloud-stream-binder-kafka and spring-cloud-stream-binder-kafka-streams. The first one is a binder implementation where it provides programming model support for writing regular Kafka producers and consumers. For the most part, you can take this same application and provide another binder (such as spring-cloud-stream-binder-rabbit) and it should work (provided that the application makes the right configuration changes). This is because the binders are the ones concerned with lower-level details of communicating to the middleware and not the app itself. Apps can largely focus on the business logic at hand. The Kafka Streams binder in Spring Cloud Stream is a binder implementation specifically built for writing streaming applications using Kafka Streams. Both Kafka binder implementations use Spring for Apache Kafka under the hood.
The rabbit binder in Spring Cloud Stream uses Spring AMQP internally.
To summarize:
Spring for Apache Kafka/Spring AMQP - lower-level foundational libraries, do not require Spring Boot.
Spring Cloud Function - Spring Cloud project providing Java 8 functional programming model, Used with Spring Boot applicaitons.
Spring Cloud Stream - Framework for event-driven applications using Spring Cloud Function. Used with Spring Boot applications.
Spring Cloud Stream Kafka/Kafka Streams - Spring Cloud Stream binder implementation using Spring for Apache Kafka. Used with Spring Boot applications.
I am a beginner in Spring mvc, Spring Boot and Spring Data JPA. I am trying to create Microservices using Spring Boot. I created a sample database CRUD operation as microservice in Spring Boot. Now I have A requirement that develop a microservice using Spring Cloud.
When I referring documentation seeing Spring tools for creating application in distributed environment. I am confused about why we are using Spring Cloud? And what is actually meant by Spring Cloud? Is there any relation with Spring mvc?
Spring Cloud is for developing some of the common patterns in distributed systems.
Spring Cloud provides tools for developers to quickly build some of the common patterns in distributed systems (e.g. configuration management, service discovery, circuit breakers, intelligent routing, micro-proxy, control bus, one-time tokens, global locks, leadership election, distributed sessions, cluster state)
Spring Cloud
For Spring Boot and Spring MVC, see this nice answer difference-between-spring-mvc-and-spring-boot
First off, thanks to the Spring team for all their work pushing this work forward!
Now that Camden.SR5 is official, I have some questions on how to correctly configure the spring cloud stream kafka binder when using Spring Boot 1.5.1.
Spring boot 1.5.1 has auto configuration for kafka and those configuration options seem to be redundant with those in the spring cloud stream kafka binder.
Do we use the core spring boot properties (spring.kafka.) or do we use (spring.cloud.stream.kafka.binder.)?
I did find this issue, but I am curious if this work will be included in the next Camden release?
https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/73
Additionally, I saw this issue reported on Stack Overflow and I believe it will also be an issue with Camden.SR5?
Failed to start bean 'inputBindingLifecycle' when using spring-boot:1.5.1 and spring-cloud-stream
Thanks
Supporting the Boot 1.5 configuration options is an issue in progress. Also, since dedicated 1.5 support is coming only with Spring Cloud Stream Chelsea release train (which is included in the Dalston release of Spring Cloud), it will be available only there.
Also, when using Spring Cloud Camden with Boot 1.5 you will need to override the Kafka dependencies as described in Failed to start bean 'inputBindingLifecycle' when using spring-boot:1.5.1 and spring-cloud-stream. This will be avoided in future versions of Spring Cloud Stream (and Spring Cloud) but only starting in the Chelsea release train of Spring Cloud Stream (and the Dalston release of Spring Cloud) - see https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/88 for details.