Has anyone tried using both Spring Cloud Function and Spring Cloud Stream together? Is there any reason this shouldn't be done? We currently use Spring Cloud Function but there are certain cases where we need to have synchronous Kafka publishing and it seems as if the only way to do that is with Spring Cloud Stream.
Thanks, Anne
Related
I was going through the https://learn.microsoft.com/en-us/azure/hdinsight/kafka/apache-kafka-producer-consumer-api looks like there is no way if we can use the Spring Cloud Stream API with the Azure Kafka. Is this understanding?
If I've to implement the Kafka Producer and Consumer still I've to use the Kafka Client API only?
I am also looking to push and read messages into Azure Kafka by using the Spring batch Approach. Can I still use KafkaItemReader and KafkaItemWriter API provided by Spring Batch with the Azure Kafka? Does it works well?
Yes, you can use the Spring libraries (Stream and Batch); they use the same kafka-clients APIs.
We have a Spring Boot Microservice that as well as having HTTP endpoints uses Spring Cloud Bus to pick up refresh events (from rabbit) and also has a Spring Cloud Stream Sink that picks up custom messages from another rabbit topic.
After updating to Spring Boot 2.4.1 and Spring Cloud 2020.0.0 everything seemed to be working until we discovered Spring Cloud Bus was no longer picking up events.
Looking into this it turned out some of the Spring Cloud Bus internal channels where not getting created.
This wasn't happening in another service that didn't have the stream functionality as well so we tested disabling that and the bus functionality then started working.
So it was obviously some sort of interference between the old style stream model and the newer Spring Cloud Bus.
After updating the our sink to use the new function model I still had issues and eventually got both to work by including the following lines in our application.yml:
spring:
cloud:
stream:
bindings.mySink-in-0.destination: mytopic
function.definition: busConsumer;mySink
So I have the following questions
Did I miss something or should there be better documentation on how stream / bus can affect each other and the migration to 2020.0.0?
Does my current configuration look correct?
It doesn't seem right to have to include busConsumer here - should the auto configuration for it not be able to 'combine it in' with any other stream config?
What's the difference between spring.cloud.stream.function.definition and spring.cloud.function.definition? I've seen both in documentation and Spring Cloud Bus seems to be also setting spring.cloud.function.definition=busConsumer
In org.springframework.cloud.stream.function.FunctionConfiguration, It does a search for #EnableBinding.
if (ObjectUtils.isEmpty(applicationContext.getBeanNamesForAnnotation(EnableBinding.class)))
If found, functional binding is disabled. See this
logger.info("Functional binding is disabled due to the presense of #EnableBinding annotation in your configuration");
After the upgrade, we need to transform our Listener classes to use functional interface in order to activate the functional binding. After that, cloud bus consumer binding will be created too.
I am aware that using Kafka Connect you can create connector sink all kafka messages to a PUBSUB topic.
I would like to know if I can use Spring Cloud Stream binders in combination with Spring Cloud Function and deploy all this to Google Functions.
In I understand correctly I can combine Spring Cloud Stream and Spring Cloud Function. Does this mean that I can use the binder from Spring Cloud Stream to actualy accept a message from Kafka ?
Google Cloud Functions uses a Trigger model to fire the functions in spring-cloud-function. Kafka as a trigger is not supported.
The workaround is to create use Kafka Connect to feed PUBSUB from a Kafka topic. You can either create your own:
https://dev.to/vtatai/make-kafka-serverless-by-connecting-it-to-google-cloud-functions-2ahh
Or use the one supplied:
https://github.com/GoogleCloudPlatform/pubsub/tree/master/kafka-connector
Using Spring for Apache Kafka, or Spring AMQP, I can achieve message pub/sub. Spring Cloud Bus uses kafka/rabbitmq to do the approximately same things, what's the differencce between them?
Spring Cloud Bus is an abstraction built on top of Spring Cloud Stream (and hence kafka and rabbitmq). It is not general purpose, but is built for sending administrative commands to multiple nodes of a service at once. For example, sending a refresh (from spring cloud commons) to all nodes of the user service. There is only one channel, where in spring cloud stream there are many. Think of it as distributed spring boot actuator.
There is a way in Spring to configure Redis queue listeners using annotations?
I would like something like Annotation-based SQS Queue Listener from Spring Cloud for AWS, but using Redis as a queue.
Looking the documentation I can't find anything that fits well for me.
This feature is already implemented in Spring or I need implement it by my own?
Spring Cloud Stream has support for redis