How to trace Kafka based events using Spring Cloud sleuth? Whatever examples I saw were for rest APIs. I am looking for kafka-clients library.
Also is it a good idea to use Spring cloud sleuth for this or should I pass my traceIds via headers manually?
From the Spring Cloud Sleuth documentation here it says that the integration is provided with Kafka Streams ( Sleuth internally uses library Brave for instrumentation). The property through which this can be enabled/disabled is spring.sleuth.messaging.kafka.streams.enabled ( true/false)
We instrument the KafkaStreams KafkaClientSupplier so that tracing
headers get injected into the Producer and Consumer's. A
KafkaStreamsTracing bean allows for further instrumentation through
additional TransformerSupplier and ProcessorSupplier methods.
For an example configuration/code you can take a look here.
Related
In the Spring Cloud website (https://spring.io/projects/spring-cloud-stream), are listed the binders options to use. And there we have the Apache Kafka and the Kafka Streams options.
What's the difference between them?
For what purpose we should choose between these two?
The Apache Kafka binder is used for basic kafka client usage consumer/producer api,
Kafka Stream binder is built upon the base apache kafka binder and adds the ability to use kafka streams api,
Kafka streams api is lightweight code libraries which gives you the functionality to manipulate data from topic/s in kafka to other topic/s in kafka , allow you to transform, enhance, filter,join, aggregate and more...
The Apache Kafka Binder implementation maps each destination to an Apache Kafka topic. The consumer group maps directly to the same Apache Kafka concept. Partitioning also maps directly to Apache Kafka partitions as well.
The binder currently uses the Apache Kafka kafka-clients version 2.3.1. This client can communicate with older brokers (see the Kafka documentation), but certain features may not be available. For example, with versions earlier than 0.11.x.x, native headers are not supported. Also, 0.11.x.x does not support the autoAddPartitions property
https://docs.spring.io/spring-cloud-stream-binder-kafka/docs/3.1.3/reference/html/spring-cloud-stream-binder-kafka.html#_apache_kafka_binder
Spring Cloud Stream includes a binder implementation designed explicitly for Apache Kafka Streams binding. With this native integration, a Spring Cloud Stream "processor" application can directly use the Apache Kafka Streams APIs in the core business logic.
Kafka Streams binder implementation builds on the foundations provided by the Spring for Apache Kafka project.
Kafka Streams binder provides binding capabilities for the three major types in Kafka Streams - KStream, KTable and GlobalKTable.
Kafka Streams applications typically follow a model in which the records are read from an inbound topic, apply business logic, and then write the transformed records to an outbound topic. Alternatively, a Processor application with no outbound destination can be defined as well.
https://docs.spring.io/spring-cloud-stream-binder-kafka/docs/3.1.3/reference/html/spring-cloud-stream-binder-kafka.html#_kafka_streams_binder
We have a Spring Boot Microservice that as well as having HTTP endpoints uses Spring Cloud Bus to pick up refresh events (from rabbit) and also has a Spring Cloud Stream Sink that picks up custom messages from another rabbit topic.
After updating to Spring Boot 2.4.1 and Spring Cloud 2020.0.0 everything seemed to be working until we discovered Spring Cloud Bus was no longer picking up events.
Looking into this it turned out some of the Spring Cloud Bus internal channels where not getting created.
This wasn't happening in another service that didn't have the stream functionality as well so we tested disabling that and the bus functionality then started working.
So it was obviously some sort of interference between the old style stream model and the newer Spring Cloud Bus.
After updating the our sink to use the new function model I still had issues and eventually got both to work by including the following lines in our application.yml:
spring:
cloud:
stream:
bindings.mySink-in-0.destination: mytopic
function.definition: busConsumer;mySink
So I have the following questions
Did I miss something or should there be better documentation on how stream / bus can affect each other and the migration to 2020.0.0?
Does my current configuration look correct?
It doesn't seem right to have to include busConsumer here - should the auto configuration for it not be able to 'combine it in' with any other stream config?
What's the difference between spring.cloud.stream.function.definition and spring.cloud.function.definition? I've seen both in documentation and Spring Cloud Bus seems to be also setting spring.cloud.function.definition=busConsumer
In org.springframework.cloud.stream.function.FunctionConfiguration, It does a search for #EnableBinding.
if (ObjectUtils.isEmpty(applicationContext.getBeanNamesForAnnotation(EnableBinding.class)))
If found, functional binding is disabled. See this
logger.info("Functional binding is disabled due to the presense of #EnableBinding annotation in your configuration");
After the upgrade, we need to transform our Listener classes to use functional interface in order to activate the functional binding. After that, cloud bus consumer binding will be created too.
I am aware that using Kafka Connect you can create connector sink all kafka messages to a PUBSUB topic.
I would like to know if I can use Spring Cloud Stream binders in combination with Spring Cloud Function and deploy all this to Google Functions.
In I understand correctly I can combine Spring Cloud Stream and Spring Cloud Function. Does this mean that I can use the binder from Spring Cloud Stream to actualy accept a message from Kafka ?
Google Cloud Functions uses a Trigger model to fire the functions in spring-cloud-function. Kafka as a trigger is not supported.
The workaround is to create use Kafka Connect to feed PUBSUB from a Kafka topic. You can either create your own:
https://dev.to/vtatai/make-kafka-serverless-by-connecting-it-to-google-cloud-functions-2ahh
Or use the one supplied:
https://github.com/GoogleCloudPlatform/pubsub/tree/master/kafka-connector
Using Spring for Apache Kafka, or Spring AMQP, I can achieve message pub/sub. Spring Cloud Bus uses kafka/rabbitmq to do the approximately same things, what's the differencce between them?
Spring Cloud Bus is an abstraction built on top of Spring Cloud Stream (and hence kafka and rabbitmq). It is not general purpose, but is built for sending administrative commands to multiple nodes of a service at once. For example, sending a refresh (from spring cloud commons) to all nodes of the user service. There is only one channel, where in spring cloud stream there are many. Think of it as distributed spring boot actuator.
I need to use Apache Kafka in my project of Micro-Services. I need my one micro-service to produce data and another to consume the same data. How can I make Kafka do the same between the two services
I would recommend you to take a look at Spring Cloud Stream as it does exactly what you need.
From docs:
Framework for building message-driven microservices. Spring Cloud Stream builds upon Spring Boot to create DevOps friendly microservice applications and Spring Integration to provide connectivity to message brokers. Spring Cloud Stream provides an opinionated configuration of message brokers, introducing the concepts of persistent pub/sub semantics, consumer groups and partitions across several middleware vendors. This opinionated configuration provides the basis to create stream processing applications.
By adding #EnableBinding to your main application, you get immediate connectivity to a message broker and by adding #StreamListener to a method, you will receive events for stream processing.