spring-cloud-stream with functional programming - apache-kafka

in my application kafka messages were consumed from multiple topics by streamlistner but i want to add functional consumer bean for one topic is this possible some topics consuming by streamlistner and some with consumer bean in same application?
when i tried topics were created only which are consumed by streamlistner but not for consumer bean?
application.yml
spring:
cloud:
function:
definition: printString
stream:
kafka:
binder:
brokers: localhost:9092
zkNodes: localhost:2181
autoCreateTopics: true
autoAddPartitions: true
bindings:
printString-in-0:
destination: function-input-topic
group: abc
consumer:
maxAttempts: 1
partitioned: true
concurrency: 2
xyz-input:
group: abc
destination: xyz-input-input
consumer:
maxAttempts: 1
partitioned: true
concurrency: 2
topic was created for xyz-input topic but not for function-input-topic
consumer bean
import java.util.function.Consumer;
#Component
public class xyz {
#Bean
Consumer<String> printString() {
return System.out::print;
}
}
kafkaConfig Interface
public interface KafkaConfig {
#Input("xyz-input")
SubscribableChannel inbound();
}

No, we tried in the past but there are subtle issues when the two programming modes collide. It is quite simple to factor out any StreamListener into a functional way (mostly removing code) and the app actually becomes much simpler.
Just get rid of KafkaConfig interface all together, get rid of #EnableBinding and you should be fine.

Related

Spring cloud stream Kafka creates a weird topic on startup

I have the following setup in application.yml for my Spring cloud stream Kafka:
spring:
cloud:
function:
definition: userBinding
stream:
kafka:
binder:
broker: localhost:9092
replicationFactor: 1
bindings:
userBinding-in-0:
destination: user
and the following consumer function to be called:
#Bean
public Consumer<Message<UserModel>> userBinding() {
return message -> {
System.out.println("Received:" + message);
};
}
For some reason as soon as I start the application it automatically creates a new topic named userBinding-in-0 instead of consuming messages from user topic!
Can you please let me know what I'm missing or setting up incorrectly?
The destination is a common property (not Kafka-specific) and shouldn't be under the kafka node.
spring:
cloud:
stream:
bindings:
userBinding-in-0:
destination: user

Spring cloud stream with kafka

Need some help in integrating kafka with spring cloud stream. The application is very simple, with 2 parts(run as separate java processes)
A consumer- puts request into RequestTopic and gets response from ResponseTopic
A producer- gets the request from the RequestTopic and puts the response back in ResponseTopic.
I have created RequestSenderChannel and ResponseReceiverChannel interfaces for consumer and RequestReceiverChannel and ResponseSenderChannel
for the producer application. both of them share the same yaml file.
As per the documentation spring.cloud.stream.bindings..destination should specify the topic to which the message is sent or received.
But when i run the application, the application creates topics as 'RequestSender', 'RequestReceiver', 'ResponseSender' and 'ResponseReceiver' in the kafka
My assumption was: since destination in the YAML file specifies only two topics 'RequestTopic' and 'ResponseTopic', it should have created those topics.
but it creates Kafka topics for attributes specified at 'spring.cloud.stream.bindings' in the YAML file.
can someone please point out the issue in the configruation/code?
public interface RequestReceiverChannel
{
String requestReceiver ="RequestReceiver";
#Input(requestReceiver)
SubscribableChannel pathQueryRequest();
}
public interface RequestSenderChannel
{
String RequestSender ="RequestSender";
#Output(RequestSender)
MessageChannel pathQueryRequestSender();
}
public interface ResponseReceiverChannel
{
String ResponseReceiver = "ResponseReceiver";
#Input(ResponseReceiver)
SubscribableChannel pceResponseServiceReceiver();
}
public interface ResponseSenderChannel
{
String ResponseSender = "ResponseSender";
#Output(ResponseSender)
MessageChannel pceResponseService();
}
'''
The YAML configuration file
spring:
cloud:
stream:
defaultBinder: kafka
bindings:
RequestSender:
binder: kafka
destination: RequestTopic
content-type: application/protobuf
group: consumergroup
ResponseSender:
binder: kafka
destination: ResponseTopic
content-type: application/protobuf
group: consumergroup
RequestReceiver:
binder: kafka
destination: RequestTopic
content-type: application/protobuf
group: consumergroup
ResponseReceiver:
binder: kafka
destination: ResponseTopic
content-type: application/protobuf
group: consumergroup
kafka:
bindings:
RequestTopic:
consumer:
autoCommitOffset: false
ResponseTopic:
consumer:
autoCommitOffset: false
binder:
brokers: ${SERVICE_KAFKA_HOST:localhost}
zkNodes: ${SERVICE_ZOOKEEPER_HOST:127.0.0.1}
defaultZkPort: ${SERVICE_ZOOKEEPER_PORT:2181}
defaultBrokerPort: ${SERVICE_KAFKA_PORT:9092}
By doing spring.cloud.stream.bindings.<binding-name>.destination=foo you are expressing desire to map binding specified by <binding-name> (e.g., RequestSender) to a broker destination named foo. If such destination does not exist it will be auto-provisioned.
So there are no issues.
That said, we've just released Horsham.RELEASE (part of cloud Hoxton.RELEASE) and we are moving away from annotation-based model you are currently using in favor of a significantly simpler functional model. You can read more about it in our release blog which also provides links to 4 posts where we elaborate and provide more examples on functional programming paradigm.

Spring Cloud #StreamListener consumer not registering CONSUMER-ID, HOST and CLIENT-ID in consumer group

We have a spring cloud consumer to read message from one kafka topic. Following is the interface for channel
#Component
public interface CollectionStreams {
String INPUT_REPORT = "report-in";
String OUTPUT_REPORT = "report-out";
#Input(INPUT_REPORT)
SubscribableChannel inboundReport();
#Output(OUTPUT_REPORT)
MessageChannel outboundReportToJM();
}
The problem we are facing is that while listing in the consumer group “report” we are not able to see CONSUMER-ID, HOST and CLIENT-ID as expected.
[root#innolx131112 templates]# kubectl -n tmo-ccm exec kafka-test-client -- /usr/bin/kafka-consumer-groups --bootstrap-server kafka:9092 --describe -group report
Note: This will not show information about old Zookeeper-based consumers.
Consumer group 'report' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
report 3 2 3 1 - - -
report 1 4 4 0 - - -
report 2 1 1 0 - - -
report 4 2 2 0 - - -
report 0 2 2 0 - - -
By the way we are running our application as well as Kafka in Kubernetes.
Due to this issue we are not able to multiple POD of our application as all PODs
Following is the interface for channel
#Component
public interface CollectionStreams {
String INPUT_REPORT = "report-in";
String OUTPUT_REPORT = "report-out";
#Input(INPUT_REPORT)
SubscribableChannel inboundReport();
#Output(OUTPUT_REPORT)
MessageChannel outboundReportToJM();
}
And we have define the method as follows to read message from topic.
#StreamListener(CollectionStreams.INPUT_REPORT)
//public void handleMessage(#Payload MessageT message) {
public void handleMessage(#Payload MessageT message, #Headers MessageHeaders msg) {
Following is configuration yaml
**
cloud:
stream:
kafka:
binder:
brokers: kafka
autoCreateTopics: false
bindings:
report-in:
consumer:
autoCommitOffset: false
autoCommit: false
auto-offset-reset: earliest
autoCommitOnError: false
resetOffsets: false
autoRebalanceEnabled: false
ackEachRecord: false
bindings:
report-in:
destination: report
contentType: application/json
group: report
consumer:
concurrency: 5
partitioned: true
report-out:
destination: jobmanager
contentType: application/json
group: jobmanager
producer:
autoAddPartitions: true
**
We also have another consumer for which we have not set any kafka related consumer props and surprisingly those consumers are registering themselves properly.
Config:
cloud:
stream:
kafka:
binder:
autoCreateTopics: false
brokers: kafka
bindings:
parse-in:
destination: parser
contentType: application/json
group: parser
consumer:
concurrency: 3
partitioned: true
parse-out:
destination: jobmanager
contentType: application/json
group: jobmanager
producer:
partitionKeyExpression: headers['contentType']
autoAddPartitions: true
And describe command output as below
[root#innolx131112 shyama]# kubectl -n tmo-ccm exec kafka-test-client -- /usr/bin/kafka-consumer-groups --bootstrap-server kafka:9092 --describe -group parser
Note: This will not show information about old Zookeeper-based consumers.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
parser 0 14 14 0 consumer-3-cb99e45e-21b3-4efb-ac7b-4f642a9486be /192.168.0.26 consumer-3
parser 1 13 13 0 consumer-3-cb99e45e-21b3-4efb-ac7b-4f642a9486be /192.168.0.26 consumer-3
parser 2 15 15 0 consumer-4-bec1e4af-d771-47fe-83ad-3440f3a6d4bd /192.168.0.26 consumer-4
parser 3 15 15 0 consumer-4-bec1e4af-d771-47fe-83ad-3440f3a6d4bd /192.168.0.26 consumer-4
parser 4 12 12 0 consumer-5-b9ac7e36-58cf-40cb-b37d-a0fa092a0d56 /192.168.0.26 consumer-5
Is it that as we are providing kafka related props for 1st consumer(report consumer), its not able to register???
Consumer group 'report' has no active members.
This simply means your app wasn't running when you entered the command.
That info is transient and not retained when the app stops.
The partitions might be assigned to a different instance next time.
EDIT
whenever we are instantiating more consumers using same group they are processing the same old already processed by old consumers
Well, looking at your configuration more carefully, that's exactly what you asked for...
autoRebalanceEnabled: false
... means that Kafka will not use group management and the partitions will be allocated by Spring Cloud stream.
autoCommitOffset: false
means that Spring will not commit any offsets and that is the responsibility of your application. If you don't commit the offset, you will get the behavior you observe.

How to configure multiple kafka consumer in application.yml file

Actually i have a springboot based micro-service , and i have used kafka to produce/consume data from different system.
Now my question is i have two different topics and based on topics i have two different consumer classes to consume data,
how to define multiple consumer properties in application.yml file ?
I configured for one consumer in application.yml like below :-
spring:
kafka:
consumer:
bootstrapservers: http://199.968.98.101:9092
group-id: groupid-QA-02
auto-offset-reset: latest
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
I am using #KafkaListener in my consumer classes
example of consumer method which i used in code
#KafkaListener(topics = "${app.topic.b2b_tf_ta_req}", groupId = "${app.topic.groupoId}")
public void consume(String message) throws Exception {
}
As far as I know bootstrap-servers accept comma separated list of servers
i.e. if you set it to server1:9092,server2:9092 kafka should connect to all of them

spring cloud stream kafka

I've built a producer spring cloud stream app and kafka as binder. Here is the application.yml:
spring:
cloud:
stream:
instanceCount : 1
bindings:
output:
destination: topic-sink
producer:
partitionSelectorClass: com.partition.CustomPartition
partitionCount: 1
...
I have two instances (same app running on a single jvm) as consumers. Here is the application.yml:
spring:
cloud:
stream:
bindings:
input:
destination: topic-sink
group: hdfs-sink
consumer:
partitioned: true
...
My understanding of kafka groups is that messages will be consumed only once, for those consumers in same group. Let's say, if the producer app produces messages A, B and there are two consumer apps in the same group, message A will be read by consumer 1 and messages B, C will be read by consumer 2. However, my consumers are consuming same messages. Are my assumptions wrong?
I got the solution, thanks Arek. For 1 partition and 1 consumer.
I share the solution for producer\consumer in spring cloud stream app.
Producer:
spring:
cloud:
stream:
instanceCount : 1
bindings:
output:
destination: topic-sink
producer:
partitionSelectorClass: com.partition.CustomPartition
partitionCount: 1
Consumer:
spring:
cloud:
stream:
instanceIndex: 0 #between 0 and instanceCount - 1
instanceCount: 1
bindings:
input:
destination: topic-sink
group: hdfs-sink
consumer:
partitioned: true
kafka:
binder:
autoAddPartitions: true